Email is universal; it has a small handful of protocols that are spoken by practically every mail client and server that are in use today. And, it happens to be one of the simplest things to migrate.
Access Level
It’s important to consider first how much access you have into your mail servers, source and target. If you happen to be moving mail between servers at which you have root access, and are using the right storage technologies, a simple rsync will do very well.
For migrations between Microsoft Exchange servers, including Office 365 or other hosted exchange providers, it is best to have administrator access to your domain to leverage certain tools, since this will encompass a lot more features of these products over other more basic syncs, like calendars and contacts.
At a bare minimum, you will need to at least have enough access to the mail account to view mail on a mail client using IMAP. In which case, it is likely that you will also have access to the target email account with IMAP. These two access levels together are all that are necessary to perform a basic mail migration.
File Migration
There are a couple of different storage methods for email server side, and luckily the most popular one is very straightforward. No, I’m not talking about your .pst file from Outlook. The two formats I encounter most frequently are mbox and maildir.
The mbox format is older and much less common, since its basically putting all your eggs in one basket. A single database file stores all of the mail for a single mailbox, generally organized by domain. These files have all of the text of the emails, along with headers and other information, concatenated together. Which is nice, but if that single file gets corrupted, there goes all the mail. For very large mailboxes, during a resync, the entire mbox file must be recopied, extending the time of the sync significantly. The changed portions cannot be updated alone.
It is very likely that any new server will be using maildir to store mail instead. Maildir does what it sounds like: it stores mail in directories. Even better, the mail is stored plaintext, so you can easily check their contents from the command line and manipulate them for sorting. Maildir is generally one directory per mail account, organized again by domain, and subfolders for the folders set up by the user to store mail. Each folder then has a ‘cur’, ‘new’, and ‘tmp’ folder to organize incoming mail, and files are named based on delivery time and message state (e.g. read or unread).
In both cases, if both servers are using the same format, you can easily rsync the files between systems. Just make sure you set up the mailboxes on the target before you bring the files over, and ensure that the permissions and ownerships of the mail files match everything else on the target system.
cPanel File Migration
The mail for an individual cPanel account is stored in /home/$username/mail/$domain/$mailbox/, while metadata for the account (mailbox passwords and quotas) is stored in /home/$username/etc/$domain/. If migrating a full domain to another account or server, copying these two folders and fixing their ownerships is sufficient to enact the migration. In addition, there are a few bonus files to look at:
/home/$username/.autorespond*/home/$username/.spamassassin*/home/$username/.sqmail*/home/$username/.cphorde//etc/valiases/$domain
These are, respectively, autoresponders, spamassassin settings, squirrelmail settings, horde settings, and forwarders. Each of these metadata files will add to the completeness of your mail migration, making a smoother experience for the clients when they start using the new server.
IMAP Migration
If you don’t have file level access, or the mail storage types are disparate (such as using mbox, or whatever Exchange uses), then you will likely need to resort to a sync of IMAP data. This is the most common mail migration method I execute, since it does not care about file level access, storage method, or even the OS of either host. As long as the source and target both serve IMAP (which any decent mail app should), you can copy mail.
It’s important to note that the IMAP protocol (internet message access protocol… protocol) will only speak mail. It does not speak address books, or contacts, or calendars, or forwarders, or anything else. It only knows how to provide mail messages. If you need any of that other stuff, you’ll have to bring it along manually later.
In order to move mail with IMAP, you’ll need to know the username and password of each and every mailbox you would like to copy. Every mailbox. 200 mailboxes? 200 usernames and passwords.
One option for transferring the mails is to set up each mailbox separately in a desktop mail client, like Thunderbird, and then drag and drop mail folders from one account to the other. This is pretty easy, because you need to set up the account in your mail client anyway, and you also don’t need to install any additional tools.
But, I do like using additional tools. Especially if they are really really good at what they do. Like imapsync.
Imapsync
Imapsync is a perl program written by Giles Larimal, which will connect over IMAP to two mailboxes, and recursively copy mail and folder structure from one to the other. It is entirely indispensible when copying mail. It’s available for install via yum:
yum install epel-release
yum install imapsyncThis should net you version 1.727 on CentOS 7. It should be a complete package, meaning there are no other prerequisites to install, but you can confirm this by just running the program and its tests:
imapsync
imapsync --testsliveIf you see connections and some dummy mail flowing around, you’re good to go! If you are unable to connect out for the tests, likely you need to open a few ports. Port 143 and port 993 will need to be accessible inbound and outbound to the server you are running imapsync on.
That’s right, you don’t even have to run imapsync on the server you are migrating to or from! This can be done entirely from your workstation, assuming you can fulfill the perl requirements.
In the category of drawbacks to imapsync, there aren’t many, but my main gripes are its verbosity and its complexity of command line entry. It can be difficult to see exactly what’s going on while the sync is running, but you can track the exit code to determine success or failure. The script will also output the mail folder and file being synced, as well as an estimate on completion, and a comparison of folders before and after the task.
The essential command line structure looks like this:
imapsync --host1 mail.hostname.com --host2 mail.hostname2.com --user1 username@domain.com --user2 username@domain.com --password1 secr3tskw1rr3l --password2 secr3tskw1rr3lIt’s a bit chewy. Don’t get me wrong; the arguments are VERY powerful, and it allows you to sync between different boxes on the same server, the same box on different servers, different boxes on different servers, and mix up SSL or non-SSL connections, as well as perform dry runs or connection tests.
Additionally, there are flags to control what mail is copied over, such as skipping messages over or under a certain size, using regex to change the folder names, deleting empty folders, or using different prefixes or separators on nonstandard servers. A quick imapsync --help will show you what I mean.
When syncing, I try to use --ssl1/2 whenever possible for secure connections, and I always do a --justlogin or --dry to ensure that the credentials I was given work, and that the connection can be established.
imapsync --host1 mail.hostname.com --host2 mail.hostname2.com --user1 username@domain.com --user2 username@domain.com --password1 secr3tskw1rr3l --password2 secr3tskw1rr3l --ssl1 --ssl2 --justloginImapsync Online
If running from the command line is not really your thing, and you only have one or two accounts to sync (and you don’t mind typing in your passwords online), then you can use the online version of imapsync, provided by the same developer.
This is very convenient if you will only be doing a small bit of syncing, as you don’t need to open ports, install any perl modules, or formulate a command line string. The web form will output the same info from imapsync as you would get from the standard command, because this is only a wrapper for executing imapsync as CGI.
If you don’t really like the idea of adding your mail passwords to an online form, I don’t blame you in the least. That’s why this online form can also be self-hosted. The instructions are a bit more in-depth than those for plain ol’ imapsync, but it can potentially be re-skinned as a tool for on-boarding your own customers, since the license for the tool is essentially WTFPL.
Finishing a Mail Migration
Mail delivery on the internet requires the orchestration of dozens of different servers, and an update in DNS MX records cannot reliably guarantee that all mail will instantly start being delivered on the target.
Fortunately, the old server shouldn’t care, and will receive any mail that straggles towards it after the DNS change. Then, a few days later, the sync method can be re-run to catch any mails that were inadvertently delivered to the old server, before the old machine is disabled.
Note that this will not work with mbox-style storage, since an rsync of this file would likely delete the entire file on the target and replace it with the old database, so anything new delivered to target would be lost. Only do re-syncs on maildir storage or IMAP-level syncs.
