Oregon State University

ONID - OSU Network ID

ONID Cyrus Migration

On August 30th, 2003 ONID migrated from standard unix mbox format mailboxes (served by UW-IMAP) to Cyrus v2.1.15 on new hardware.

During the migration, we nfs mounted /var/spool/mail from the old mail server onto the new mail server for the purpose of copying the old inboxes. We nfs mount user home directories onto the Cyrus mail server so we can read the user's .sieve file, and we were also able to copy mail from each user's ~/mail directory during the migration this way.

It is worth noting that these scripts do not migrate IMAP flags on messages, including the Seen state of the messages. We decided that it wasn't worth the effort of trying to figure out how to do this.

If you have questions about the migration or these scripts, feel free to contact me (Andy Morgan) at morgan at orst dot edu.

These are the steps we performed during the migration (on the new Cyrus mail server, unless otherwise specified):

  1. Prevent all access and updates to the mail files on the old server by stopping UW-IMAP imapd, sendmail, and local shell access. Mail will queue up on remote servers during this time, so no mail will be lost. Also, point the MX record to the new mail server (the campus mail relays in our case).
  2. Start Cyrus IMAP on the new server, but only allow IMAP access from the localhost using tcpwrappers entry in /etc/hosts.deny. Don't accept LMTP connections yet (block using tcpwrappers as well).
  3. Run batchmigrate.pl. This took about 8 hours for our 35,000 accounts and about 130GB of mail.
  4. Run "su cyrus -c '/usr/local/cyrus/bin/quota -f'" to fix all the quota information on the system.
  5. Run batchsieve.pl to migrate existing .procmailrc, .vacation.msg, and .forward files into a new .sieve file.
  6. Run batchlock.pl to change the ownership of each user's ~/mail directory to root and set the permissions for access by only root. We didn't want to delete this directory just yet, in case we needed to retrieve the data later, but we don't want user's to access this directory thinking that their mail is still stored there.
  7. Remove the tcpwrappers blocks in /etc/hosts.deny to start accepting IMAP and LMTP connections on the new server.

The individual scripts used are described below. These scripts, and some other handy administration tools, are available at http://oregonstate.edu/~morgan/cyrus/.


batchmigrate.pl

This script is a wrapper around migrate_user_mail.pl. It takes one argument, a file containing a list of unix usernames (one per line), and calls migrate_user_mail.pl for each username. It also captures the output of migrate_user_mail.pl into a log file for each username.


migrate_user_mail.pl

This script is does all the real migration work. It migrates the user's inbox and all mail in their ~/mail directory. Several other scripts and utilities are called by this script to do the low-level copying of mail: filter.pl, formail (from the procmail package), and cpmsg.pl. This script also sets the quota for each user.


filter.pl

This script takes a mailbox on stdin, removes the first message if it is a FOLDER INTERAL DATA message, and prints the mailbox out on stdout.


cpmsg.pl

This script gets called once for each message by formail. It takes one argument, which is the Cyrus maildir to put the message into. It takes the message on stdin from formail, appends a CR-LF to each line, and writes it out into the maildir.


batchsieve.pl

This script is a wrapper around migrate_sieve.pl. It takes one argument, a file containing a list of unix usernames (one per line), and calls migrate_sieve.pl for each username. It also captures the output of migrate_sieve.pl into a log file for each username.


migrate_sieve.pl

This script migrates a user's .procmailrc, .vacation.msg, and .forward files into a .sieve file. It takes one argument, the username to migrate. It does a reasonable job of handling all the strange possibilities that can happen in .procmailrc and .forward files, but you need to check the log files to see what users it was not able to migrate.


batchlock.pl

This script is a wrapper around lock_user_maildir.pl. It takes one argument, a file containing a list of unix usernames (one per line), and calls lock_user_maildir.pl for each username. It also captures the output of lock_user_maildir.pl into a log file for each username.


lock_user_maildir.pl

This script locks down a user's ~/mail directory. It takes one argument, the username to lock down. Ownership of all subdirectories and files is changed to root. Permissions are set to 600 for files and 700 for directories.


Enterprise Computing Services, Oregon State University, Corvallis, OR 97331.
Contact Support - (1 541 737 8787)
Copyright Oregon State University | Disclaimer.