Migrate mail from one server to another with imapsync

Have you been in the situation where you needed to move everyone’s mail from one provider to another? It’s a pain to do using the mail client one at a time, but with this guide, you’ll be able to create a csv with everyone’s info and run one command to fire it all off. All you need is a Linux machine to do it.

First, you’ll need to install the epel repo (skip if you already have it):

CentOS 5 (as root):

wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-5.rpm
rpm -Uvh remi-release-5*.rpm epel-release-5*.rpm

wget http://dl.fedoraproject.org/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-5.rpm

rpm -Uvh remi-release-5*.rpm epel-release-5*.rpm

CentOS 6 (as root):

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm

rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

Now, use yum to install imapsync. It will automagically install all of the deps.

yum install imapsync

1	yum install imapsync

Now that it’s installed, let’s print out the help file and take a look at all of the usefulness of imapsync:

[root@backup BACKUP]# imapsync --help

usage: /usr/bin/imapsync [options]

Several options are mandatory. 

--dry                  : Makes imapsync doing nothing, just print what would 
                         be done without --dry.

--host1       <string> : Source or "from" imap server. Mandatory.
--port1       <int>    : Port to connect on host1. Default is 143.
--user1       <string> : User to login on host1. Mandatory.
--showpasswords        : Shows passwords on output instead of "MASKED".
                         Useful to restart a complete run by just reading a log.
--password1   <string> : Password for the user1.
--host2       <string> : "destination" imap server. Mandatory.
--port2       <int>    : Port to connect on host2. Default is 143.
--user2       <string> : User to login on host2. Mandatory.
--password2   <string> : Password for the user2.

--passfile1   <string> : Password file for the user1. It must contain the 
                         password on the first line. This option avoids to show 
                         the password on the command line like --password1 does.
--passfile2   <string> : Password file for the user2. Contains the password.
--domain1     <string> : Domain on host1 (NTLM authentication).
--domain2     <string> : Domain on host2 (NTLM authentication).
--authuser1   <string> : User to auth with on host1 (admin user). 
                         Avoid using --authmech1 SOMETHING with --authuser1.
--authuser2   <string> : User to auth with on host2 (admin user).
--proxyauth1           : Use proxyauth on host1. Requires --authuser1.
                         Required by Sun/iPlanet/Netscape IMAP servers to
                         be able to use an administrative user.
--proxyauth2           : Use proxyauth on host2. Requires --authuser2.
                         Required by Sun/iPlanet/Netscape IMAP servers to
                         be able to use an administrative user

--authmd51             : Use MD5 authentification for host1.
--authmd52             : Use MD5 authentification for host2.
--authmech1   <string> : Auth mechanism to use with host1:
                         PLAIN, LOGIN, CRAM-MD5 etc. Use UPPERCASE.
--authmech2   <string> : Auth mechanism to use with host2. See --authmech1
--ssl1                 : Use an SSL connection on host1.
--ssl2                 : Use an SSL connection on host2.
--tls1                 : Use an TLS connection on host1.
--tls2                 : Use an TLS connection on host2.
--timeout    <int>     : Connections timeout in seconds. Default is 120.
                         0 means no timeout.

--folder      <string> : Sync this folder.
--folder      <string> : and this one, etc.
--folderrec   <string> : Sync this folder recursively.
--folderrec   <string> : and this one, etc.
--include     <regex>  : Sync folders matching this regular expression
                         Blancs like in "foo bar" have to be written "foo\ bar"
--include     <regex>  : or this one, etc.
                         in case both --include --exclude options are
                         use, include is done before.
--exclude     <regex>  : Skips folders matching this regular expression
                         Several folders to avoid:
			  --exclude 'fold1|fold2|f3' skips fold1, fold2 and f3.
--exclude     <regex>  : or this one, etc.
--regextrans2 <regex>  : Apply the whole regex to each destination folders.
--regextrans2 <regex>  : and this one. etc.
                         When you play with the --regextrans2 option, first
                         add also the safe options --dry --justfolders
                         Then, when happy, remove --dry, remove --justfolders.
                         Have in mind that --regextrans2 is applied after prefix 
                         and separator inversion.

--tmpdir      <string> : Where to store temporary files and subdirectories.
                         Will be created if it doesn't exist.
			 Default is system specific, Unix is /tmp but
                         it's often small and deleted at reboot.
                         --tmpdir /var/tmp should be better.
--pidfile     <string> : The file where imapsync pid is written.
--pidfilelocking       : Abort if pidfile already exists. Usefull to avoid 
                         concurrent transfers on the same mailbox.

--prefix1     <string> : Remove prefix to all destination folders 
                         (usually INBOX. or INBOX/ or an empty string "")
                         you have to use --prefix1 if host1 imap server
                         does not have NAMESPACE capability, all other
                         cases are bad.
--prefix2     <string> : Add prefix to all host2 folders. See --prefix1
--sep1        <string> : Host1 separator in case NAMESPACE is not supported.
--sep2        <string> : Host2 separator in case NAMESPACE is not supported.

--regexmess   <regex>  : Apply the whole regex to each message before transfer.
                         Example: 's/\000/ /g' # to replace null by space.
--regexmess   <regex>  : and this one.
--regexmess   <regex>  : and this one, etc.
--regexflag   <regex>  : Apply the whole regex to each flags list.
                         Example: 's/"Junk"//g' # to remove "Junk" flag.
--regexflag   <regex>  : and this one, etc.

--delete               : Deletes messages on host1 server after a successful 
                         transfer. Option --delete has the following behavior: 
                         it marks messages as deleted with the IMAP flag 
                         \Deleted, then messages are really deleted with an 
                         EXPUNGE IMAP command.
--delete2              : Delete messages in host2 that are not in
                         host1 server. Useful for backup or pre-sync.
--delete2duplicates    : Delete messages in host2 that are duplicates.
                         Works only without --useuid since duplicates are 
                         detected with header part of each message.
--delete2folders       : Delete folders in host2 that are not in host1 server. 
                         For safety, first try it like this (it is safe):
			 --delete2folders --dry --justfolders --nofoldersizes
--delete2foldersonly   <regex>: Deleted only folders matching regex.
--delete2foldersbutnot <regex>: Do not delete folders matching regex.
                         Example: --delete2foldersbutnot "/Tasks|Contacts|Foo/"
--noexpunge            : Do not expunge messages on host1.
                         Expunge really deletes messages marked deleted.
                         Expunge is made at the beginning, on host1 only. 
                         Newly transferred messages are also expunged if 
			 option --delete is given.
                         No expunge is done on host2 account (unless --expunge2)
--expunge1             : Expunge messages on host1 after messages transfer.
--expunge2             : Expunge messages on host2 after messages transfer.
--uidexpunge2          : uidexpunge messages on the host2 account
                         that are not on the host1 account, requires --delete2

--syncinternaldates    : Sets the internal dates on host2 same as host1.
                         Turned on by default. Internal date is the date
			 a message arrived on a host (mtime).
--idatefromheader      : Sets the internal dates on host2 same as the 
                         "Date:" headers.

--maxsize     <int>    : Skip messages larger  (or equal) than <int> bytes
--minsize     <int>    : Skip messages smaller (or equal) than <int> bytes
--maxage      <int>    : Skip messages older than <int> days.
                         final stats (skipped) don't count older messages
			 see also --minage
--minage      <int>    : Skip messages newer than <int> days.
                         final stats (skipped) don't count newer messages
                         You can do (+ are the messages selected):
                         past|----maxage+++++++++++++++>now
                         past|+++++++++++++++minage---->now
                         past|----maxage+++++minage---->now (intersection)
                         past|++++minage-----maxage++++>now (union)

--search      <string> : Selects only messages returned by this IMAP SEARCH 
                         command. Applied on both sides.
--search1     <string> : Same as --search for selecting host1 messages only.
--search2     <string> : Same as --search for selecting host2 messages only.
                         --search CRIT equals --search1 CRIT --search2 CRIT

--exitwhenover   <int> : Stop syncing when total bytes transferred reached.
                         Gmail per day allows 2500000000 down 500000000 upload.

--maxlinelength <int>  : skip messages with line length longer than <int> bytes.
                         RFC 2822 says it must be no more than 1000 bytes.

--useheader   <string> : Use this header to compare messages on both sides.
                         Ex: Message-ID or Subject or Date.
--useheader   <string>   and this one, etc.

--subscribed           : Transfers subscribed folders.
--subscribe            : Subscribe to the folders transferred on the 
                         host2 that are subscribed on host1. On by default.
--subscribe_all        : Subscribe to the folders transferred on the 
                         host2 even if they are not subscribed on host1.

--nofoldersizes        : Do not calculate the size of each folder in bytes
                         and message counts. Default is to calculate them.
--nofoldersizesatend   : Do not calculate the size of each folder in bytes
                         and message counts at the end. Default is on.
--justfoldersizes      : Exit after having printed the folder sizes.

--syncacls             : Synchronises acls (Access Control Lists).
--nosyncacls           : Does not synchronize acls. This is the default.
                         Acls in IMAP are not standardized, be careful.

--usecache             : Use cache to speedup.
--nousecache           : Do not use cache. Caveat: --useuid --nousecache creates
                         duplicates on multiple runs.
--useuid               : Use uid instead of header as a criterium to recognize 
                         messages. Option --usecache is then implied unless 
                         --nousecache is used.  

--debug                : Debug mode.
--debugcontent         : Debug content of the messages transfered.
--debugflags           : Debug flags.
--debugimap1           : IMAP debug mode for host1. imap debug is very verbose.
--debugimap2           : IMAP debug mode for host2.
--debugimap            : IMAP debug mode for host1 and host2.

--version              : Print software version.
--noreleasecheck       : Do not check for new imapsync release (a http request).
--justconnect          : Just connect to both servers and print useful
                         information. Need only --host1 and --host2 options.
--justlogin            : Just login to both host1 and host2 with users 
                         credentials, then exit.
--justfolders          : Do only things about folders (ignore messages).

--help                 : print this help.

Example: to synchronize imap account "foo" on "imap.truc.org"
                    to  imap account "bar" on "imap.trac.org"
                    with foo password "secret1"
                    and  bar password "secret2"

/usr/bin/imapsync \
   --host1 imap.truc.org --user1 foo --password1 secret1 \
   --host2 imap.trac.org --user2 bar --password2 secret2

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

[root@backup BACKUP]# imapsync --help

usage: /usr/bin/imapsync [options]

Several options are mandatory.

--dry : Makes imapsync doing nothing, just print what would

be done without --dry.

--host1 <string> : Source or "from" imap server. Mandatory.

--port1 <int> : Port to connect on host1. Default is 143.

--user1 <string> : User to login on host1. Mandatory.

--showpasswords : Shows passwords on output instead of "MASKED".

Useful to restart a complete run by just reading a log.

--password1 <string> : Password for the user1.

--host2 <string> : "destination" imap server. Mandatory.

--port2 <int> : Port to connect on host2. Default is 143.

--user2 <string> : User to login on host2. Mandatory.

--password2 <string> : Password for the user2.

--passfile1 <string> : Password file for the user1. It must contain the

password on the first line. This option avoids to show

the password on the command line like --password1 does.

--passfile2 <string> : Password file for the user2. Contains the password.

--domain1 <string> : Domain on host1 (NTLM authentication).

--domain2 <string> : Domain on host2 (NTLM authentication).

--authuser1 <string> : User to auth with on host1 (admin user).

Avoid using --authmech1 SOMETHING with --authuser1.

--authuser2 <string> : User to auth with on host2 (admin user).

--proxyauth1 : Use proxyauth on host1. Requires --authuser1.

Required by Sun/iPlanet/Netscape IMAP servers to

be able to use an administrative user.

--proxyauth2 : Use proxyauth on host2. Requires --authuser2.

Required by Sun/iPlanet/Netscape IMAP servers to

be able to use an administrative user

--authmd51 : Use MD5 authentification for host1.

--authmd52 : Use MD5 authentification for host2.

--authmech1 <string> : Auth mechanism to use with host1:

PLAIN, LOGIN, CRAM-MD5 etc. Use UPPERCASE.

--authmech2 <string> : Auth mechanism to use with host2. See --authmech1

--ssl1 : Use an SSL connection on host1.

--ssl2 : Use an SSL connection on host2.

--tls1 : Use an TLS connection on host1.

--tls2 : Use an TLS connection on host2.

--timeout <int> : Connections timeout in seconds. Default is 120.

0 means no timeout.

--folder <string> : Sync this folder.

--folder <string> : and this one, etc.

--folderrec <string> : Sync this folder recursively.

--folderrec <string> : and this one, etc.

--include <regex> : Sync folders matching this regular expression

Blancs like in "foo bar" have to be written "foo\ bar"

--include <regex> : or this one, etc.

in case both --include --exclude options are

use, include is done before.

--exclude <regex> : Skips folders matching this regular expression

Several folders to avoid:

--exclude 'fold1|fold2|f3' skips fold1, fold2 and f3.

--exclude <regex> : or this one, etc.

--regextrans2 <regex> : Apply the whole regex to each destination folders.

--regextrans2 <regex> : and this one. etc.

When you play with the --regextrans2 option, first

add also the safe options --dry --justfolders

Then, when happy, remove --dry, remove --justfolders.

Have in mind that --regextrans2 is applied after prefix

and separator inversion.

--tmpdir <string> : Where to store temporary files and subdirectories.

Will be created if it doesn't exist.

Default is system specific, Unix is /tmp but

it's often small and deleted at reboot.

--tmpdir /var/tmp should be better.

--pidfile <string> : The file where imapsync pid is written.

--pidfilelocking : Abort if pidfile already exists. Usefull to avoid

concurrent transfers on the same mailbox.

--prefix1 <string> : Remove prefix to all destination folders

(usually INBOX. or INBOX/ or an empty string "")

you have to use --prefix1 if host1 imap server

does not have NAMESPACE capability, all other

cases are bad.

--prefix2 <string> : Add prefix to all host2 folders. See --prefix1

--sep1 <string> : Host1 separator in case NAMESPACE is not supported.

--sep2 <string> : Host2 separator in case NAMESPACE is not supported.

--regexmess <regex> : Apply the whole regex to each message before transfer.

Example: 's/\000/ /g' # to replace null by space.

--regexmess <regex> : and this one.

--regexmess <regex> : and this one, etc.

--regexflag <regex> : Apply the whole regex to each flags list.

Example: 's/"Junk"//g' # to remove "Junk" flag.

--regexflag <regex> : and this one, etc.

--delete : Deletes messages on host1 server after a successful

transfer. Option --delete has the following behavior:

it marks messages as deleted with the IMAP flag

\Deleted, then messages are really deleted with an

EXPUNGE IMAP command.

--delete2 : Delete messages in host2 that are not in

host1 server. Useful for backup or pre-sync.

--delete2duplicates : Delete messages in host2 that are duplicates.

Works only without --useuid since duplicates are

detected with header part of each message.

--delete2folders : Delete folders in host2 that are not in host1 server.

For safety, first try it like this (it is safe):

--delete2folders --dry --justfolders --nofoldersizes

--delete2foldersonly <regex>: Deleted only folders matching regex.

--delete2foldersbutnot <regex>: Do not delete folders matching regex.

Example: --delete2foldersbutnot "/Tasks|Contacts|Foo/"

--noexpunge : Do not expunge messages on host1.

Expunge really deletes messages marked deleted.

Expunge is made at the beginning, on host1 only.

Newly transferred messages are also expunged if

option --delete is given.

No expunge is done on host2 account (unless --expunge2)

--expunge1 : Expunge messages on host1 after messages transfer.

--expunge2 : Expunge messages on host2 after messages transfer.

--uidexpunge2 : uidexpunge messages on the host2 account

that are not on the host1 account, requires --delete2

--syncinternaldates : Sets the internal dates on host2 same as host1.

Turned on by default. Internal date is the date

a message arrived on a host (mtime).

--idatefromheader : Sets the internal dates on host2 same as the

"Date:" headers.

--maxsize <int> : Skip messages larger (or equal) than <int> bytes

--minsize <int> : Skip messages smaller (or equal) than <int> bytes

--maxage <int> : Skip messages older than <int> days.

final stats (skipped) don't count older messages

see also --minage

--minage <int> : Skip messages newer than <int> days.

final stats (skipped) don't count newer messages

You can do (+ are the messages selected):

past|----maxage+++++++++++++++>now

past|+++++++++++++++minage---->now

past|----maxage+++++minage---->now (intersection)

past|++++minage-----maxage++++>now (union)

--search <string> : Selects only messages returned by this IMAP SEARCH

command. Applied on both sides.

--search1 <string> : Same as --search for selecting host1 messages only.

--search2 <string> : Same as --search for selecting host2 messages only.

--search CRIT equals --search1 CRIT --search2 CRIT

--exitwhenover <int> : Stop syncing when total bytes transferred reached.

Gmail per day allows 2500000000 down 500000000 upload.

--maxlinelength <int> : skip messages with line length longer than <int> bytes.

RFC 2822 says it must be no more than 1000 bytes.

--useheader <string> : Use this header to compare messages on both sides.

Ex: Message-ID or Subject or Date.

--useheader <string> and this one, etc.

--subscribed : Transfers subscribed folders.

--subscribe : Subscribe to the folders transferred on the

host2 that are subscribed on host1. On by default.

--subscribe_all : Subscribe to the folders transferred on the

host2 even if they are not subscribed on host1.

--nofoldersizes : Do not calculate the size of each folder in bytes

and message counts. Default is to calculate them.

--nofoldersizesatend : Do not calculate the size of each folder in bytes

and message counts at the end. Default is on.

--justfoldersizes : Exit after having printed the folder sizes.

--syncacls : Synchronises acls (Access Control Lists).

--nosyncacls : Does not synchronize acls. This is the default.

Acls in IMAP are not standardized, be careful.

--usecache : Use cache to speedup.

--nousecache : Do not use cache. Caveat: --useuid --nousecache creates

duplicates on multiple runs.

--useuid : Use uid instead of header as a criterium to recognize

messages. Option --usecache is then implied unless

--nousecache is used.

--debug : Debug mode.

--debugcontent : Debug content of the messages transfered.

--debugflags : Debug flags.

--debugimap1 : IMAP debug mode for host1. imap debug is very verbose.

--debugimap2 : IMAP debug mode for host2.

--debugimap : IMAP debug mode for host1 and host2.

--version : Print software version.

--noreleasecheck : Do not check for new imapsync release (a http request).

--justconnect : Just connect to both servers and print useful

information. Need only --host1 and --host2 options.

--justlogin : Just login to both host1 and host2 with users

credentials, then exit.

--justfolders : Do only things about folders (ignore messages).

--help : print this help.

Example: to synchronize imap account "foo" on "imap.truc.org"

to imap account "bar" on "imap.trac.org"

with foo password "secret1"

and bar password "secret2"

/usr/bin/imapsync \

--host1 imap.truc.org --user1 foo --password1 secret1 \

--host2 imap.trac.org --user2 bar --password2 secret2

Now let’s get to business and start a sync. Let’s just do one user first before we jump into doing multiple at once.

Our info for this task:
Old: server=old.mailserver.com, user=test@domain.com, pass=mypasswd
New: server=new.mailserver.com, user=test@domain.com, pass=mypasswd

Here’s the command:

/usr/bin/imapsync \
   --host1 old.mailserver.com --user1 test@domain.com --password1 mypasswd \
   --host2 new.mailserver.com --user2 test@domain.com --password2 mypasswd

/usr/bin/imapsync \

--host1 old.mailserver.com --user1 test@domain.com --password1 mypasswd \

--host2 new.mailserver.com --user2 test@domain.com --password2 mypasswd

Let’s migrate multiple people at once now!

Create a file full of info called ‘file.txt’. An example is here

test@domain.com;mypasswd;test@domain.com;mypasswd
test2@domain.com;hispasswd;test2@domain.com;hispasswd
test3@domain.com;herpasswd;test3@domain.com;herpasswd

test@domain.com;mypasswd;test@domain.com;mypasswd

test2@domain.com;hispasswd;test2@domain.com;hispasswd

test3@domain.com;herpasswd;test3@domain.com;herpasswd

Now, paste this into sync_loop_unix.sh

#!/bin/sh
#
# $Id: sync_loop_unix.sh,v 1.2 2012/12/23 08:02:46 gilles Exp gilles $

# Example for imapsync massive migration on Unix systems.
# 
# Data is supposed to be in file.txt in the following format
#user001_1;password001_1;user001_2;password001_2
#...
# Separator is character semi-colon ; it can be changed by any character changing IFS=';'
# Each data line contains 4 columns, columns are parameters for --user1 --password1 --user2 --password2
#
# Replace "imap.side1.org" and "imap.side2.org" with your own hostname values

# This loop will also create a log file called LOG/log_${u2}_$NOW.txt for each account transfer
# where u2 is just a variable containing the user2 account name, and NOW is the current date_time

mkdir -p LOG

{ while IFS=';' read  u1 p1 u2 p2
    do 
         { echo "$u1" | egrep "^#" ; } > /dev/null && continue
         NOW=`date +%Y_%m_%d_%H_%M_%S` 
         echo syncing to user "$u2"
         imapsync --host1 imap.side1.org --user1 "$u1" --password1 "$p1" \
                  --host2 imap.side2.org --user2 "$u2" --password2 "$p2" \
                  > LOG/log_${u2}_$NOW.txt 2>&1
    done 
} < file.txt

#!/bin/sh

# $Id: sync_loop_unix.sh,v 1.2 2012/12/23 08:02:46 gilles Exp gilles $

# Example for imapsync massive migration on Unix systems.

# Data is supposed to be in file.txt in the following format

#user001_1;password001_1;user001_2;password001_2

#...

# Separator is character semi-colon ; it can be changed by any character changing IFS=';'

# Each data line contains 4 columns, columns are parameters for --user1 --password1 --user2 --password2

# Replace "imap.side1.org" and "imap.side2.org" with your own hostname values

# This loop will also create a log file called LOG/log_${u2}_$NOW.txt for each account transfer

# where u2 is just a variable containing the user2 account name, and NOW is the current date_time

mkdir -p LOG

{ while IFS=';' read u1 p1 u2 p2

{ echo "$u1" | egrep "^#" ; } > /dev/null && continue

NOW=`date +%Y_%m_%d_%H_%M_%S`

echo syncing to user "$u2"

imapsync --host1 imap.side1.org --user1 "$u1" --password1 "$p1" \

--host2 imap.side2.org --user2 "$u2" --password2 "$p2" \

> LOG/log_${u2}_$NOW.txt 2>&1

done

} < file.txt

(Make sure to change the hosts in the sync_loop_unix.sh file)

Make it executable:

chmod +x sync_loop_unix.sh

1	chmod +x sync_loop_unix.sh

Now run it!

./sync_loop_unix.sh

1	./sync_loop_unix.sh

You can add as many accounts as you want in the csv file.

Read more about imapsync on their official page: http://imapsync.lamiral.info/

2 responses to Migrate mail from one server to another with imapsync

Godislife January 11, 2015 at 9:58 pm

thanks for you tutorial!

Michel February 16, 2015 at 3:50 am

Thank you very much!

Hary
Many Thanks Brother
web hoster
How would you add a c class to CSF?
Automysqlbackup fix error "Skipping the data of table mysql.event. Specify the –events option explicitly" | DL-UAT
[…] found that its because of not ...

Related posts from Linux Brigade!

2 responses to Migrate mail from one server to another with imapsync

Leave a Reply Cancel reply

Categories

Migrate mail from one server to another with imapsync

Related posts from Linux Brigade!

2 responses to Migrate mail from one server to another with imapsync

Leave a Reply Cancel reply

Share…

Categories