Installing The Cyrus Murder
Note that Cyrus Murder is still relatively young in the grand scheme of
things, and if you choose to deploy you are doing so at your own risk.
Many of the failure modes can be difficult to track without a detailed
understanding of the mupdate protocol and IMAP in general, and thus
even considering a deployment is not for the faint at heart.
Sites which think they need the Murder functionality but do not need a
uniform namespace (no shared mailboxes) should consider other IMAP
proxy solutions.
At the same time, we are using it successfully in production at
Carnegie Mellon.
Introduction & Assumptions
This document is intended to be a guide to the configuration of a Cyrus
IMAP Aggregator, aka Cyrus Murder. It is recommended that you review
this document to be familliar with the concepts at work. This document
is a work in progress and is at this point incomplete.
This document assumes that you have successfully been able to setup
atleast one Cyrus IMAP server. This server will become your first
backend server. It also assumes that you are familliar with the
administration and day to day operations of the Cyrus IMAP server and
the SASL authentication library. If you feel uncomfortable with this,
please refer to the rest of the documentation first.
There is a diagram that shows the interactions of the various
components of the Cyrus Murder which may be helpful in understanding
the "big picture".
Installation
You will need to build Cyrus IMAPd with the --enable-murder configure
option. This builds the proxyds and the associated utilities.
Requirements
* Atleast one Cyrus IMAP server. If there are more than one, their
name spaces must not conflict. That is, all the mailbox names must
be unique (or in different namespaces)
* Atleast one machine that will become the first Frontend Server.
* One machine to become the MUPDATE master server. This can be the
same as one of your frontend servers.
Configuring the MUPDATE Master
The mupdate master server needs to be running the mupdate service in
master mode. Note that you can have the MUPDATE master be one of your
frontend machines, just do not configure a slave mupdate process on
this machine.
To configure an mupdate master, you will want a cyrus.conf that
includes a line similar to the following in the SERVICES section:
mupdate cmd="/usr/cyrus/bin/mupdate -m" listen=3905 prefork=1
Note the "-m" option to tell mupdate that it should start in master
mode.
You will also need to configure atleast a skeleton imapd.conf that
defines the configdirectory, a bogus partition-default and the admins
that can authenticate to the server. Note that slave mupdate servers as
well as the backend servers will need to be able to authenticate as
admins on the master. Here is a very simple imapd.conf for a master
server:
configdirectory: /imap/conf
partition-default: /tmp
admins: mupdateslave1 backend1
You will also need to configure SASL to properly allow authentication
in your enviornment.
Setting up the backends to push changes to the MUPDATE Master
On the backends, configuration to be a part of a murder is easy. You
just need to configure the backend to be a part of the murder. To do
this, set the mupdate_server option in imapd.conf. Depending on what
authentication mechanisms you are using, you may also want to set some
or all of the following:
* mupdate_username
* mupdate_authname
* mupdate_realm
* mupdate_password
Once these settings are successfully made, any mailbox operation on the
backend will be sent to the mupdate master for confirmation and entry
into the mupdate database.
You will also want to configure atleast one user/group using the
proxyservers imapd.conf option. This user should not be an
administrator, since this means that anyone who can get ahold of your
proxy servers now has full administrative control on your backend.
Example:
admins: cyrus
proxyservers: murder
Keep in mind that you will need to create the proxy user(s) and be sure
that they can authenticate to the backend as well.
Importing the database from the backend
Importing the current mailboxes database is easy, as there is a
ctl_mboxlist option to do so. To do the first synchronization, simply
change to the cyrus user, and issue a ctl_mboxlist -m.
Note that you may wish to issue a ctl_mboxlist -mw first to be sure you
understand all the operations that this command will perform, since it
does require that all mailboxes are unique in the murder namespace. If
everything is configured properly, the mailbox database of the current
host will dump to the mupdate master. If there are problems, the most
likely cause is a misconfiguration of the authentication settings, or
that mupdate might not be running on the master. Using mupdatetest may
be helpful in this case (it establishes an authenticated connection to
the mupdate server, if it can).
It is also useful to have the backends automatically resync the state
of their local mailboxes database with the master on start up. You can
configure this by adding the following to the START section of
cyrus.conf on the backends:
mupdatepush cmd="ctl_mboxlist -m"
This will perform synchronization with the mupdate master each time the
backend restarts, bringing the mupdate database up to date with the
contents of the backend (and performing ACTIVATE and DELETES as needed
to do so).
Warning: If somehow a mailbox exists on two (or more) backend servers,
each time one of them synchronizes its database that backend server
will become authoritative. Though this should not happen during normal
operation of the murder (because of the consistancy guarantees of the
MUPDATE protocol, and the fact that mailbox operations are denied if
the mupdate master is down), it is possible when first creating the
mupdate database or when bringing a new backend server into the murder.
Configuring the frontends
Configuring the frontends is a two step process. First, you want to set
mupdate_server (and friends) as you did for the backends above.
However, because the frontends only talk to the mupdate master via a
slave running on the local machine, you will also need to set up a
slave on the same machine, in the SERVICES section of cyrus.conf, like
so:
# mupdate database service - must prefork atleast 1
mupdate cmd="mupdate" listen=3905 prefork=1
Note that as this is a threaded service, you must prefork atleast 1 of
them so that the database can be synchronized at startup. Otherwise,
the service will not start running until after you recieve an mupdate
client connection to the slave (which is not a recommended
configuration at this point).
You will also want to change all of your imapd entries to be proxyd,
and all of your lmtpd entries to be lmtpproxyd. That is, you will
probably have a SERVICES section that looks more like this now:
mupdate cmd="/usr/cyrus/bin/mupdate" listen=3905 prefork=1
imap cmd="proxyd" listen="imap" prefork=5
imaps cmd="proxyd -s" listen="imaps" prefork=1
pop3 cmd="pop3d" listen="pop3" prefork=0
pop3s cmd="pop3d -s" listen="pop3s" prefork=0
kpop cmd="pop3d -k" listen="kpop" prefork=0
nntp cmd="nntpd" listen="nntp" prefork=0
nntps cmd="nntpd -s" listen="nntps" prefork=0
sieve cmd="timsieved" listen="sieve" prefork=0
lmtp cmd="lmtpproxyd" listen="/var/imap/socket/lmtp" prefork=0
Note that timsieved does not need a proxy daemon, the managesieve
protocol deals with the murder with referrals to the backends
internally.
Additionally, you will need entries in imapd.conf to indicate the proxy
auth name and passwords (if you are using a SASL mechanism that
requires them) to the backends, for example, if your backends are
mail1.andrew.cmu.edu and mail2.andrew.cmu.edu with passwords of foo and
bar, and an auth name of murder:
mail1_password: foo
mail2_password: bar
proxy_authname: murder
If your SASL mechanism does not require authnames or passwords (e.g.
KERBEROS_V4), then this is not required. Note that we used the same
authname as the configured in the proxyservers line in the backend's
imapd.conf above.
When you start master on the frontend, a local mailboxes database
should automatically synchronize itself with the contents of the
mupdate master, and you should be ready to go. Your clients should
connect to the frontends, and the frontends will proxy or refer as
applicable to the blackend servers.
Additional backend configuration
If your authentication system requires usernames, passwords, etc, to
authenticate (e.g. it isn't Kerberos), then you will also need to
specify proxy_authname (and friends) in the backend imapd.confs as
well. This is so that the backends can authenticate to eachother to
facilitate maibox moves. (Backend machines will need to be full
admins).
Delivering mail
To deliver mail to your Murder, configure your MTA just as you did
before, but instead of connecting directly to lmtpd, it should connect
to lmtpproxyd. You can connect to the lmtpproxyd running on the
frontend machines, or you can install master and lmtpproxyd on your
SMTP servers.
Administration
Keeping the database synced
Consistancy in the database is maintained by pushing the current status
of the backends to the master, and having the frontends stay up to date
with the master's database. Since the frontends resync themselves
entirely when they startup, downtime should not at all be a problem.
(While they are up they should be continously recieving database
updates, as well as when they lose connection to the master, they will
try to reconnect and resync their database upon reconnection)
Provided that the namespace of the backend servers is kept discrete
(with no mailboxes existing on the same server), it is not a big deal
to resync the mupdate master using ctl_mboxlist -m. If two servers do
have the same mailbox, this will need to be resolved before database
consistancy can be guranteed.
Moving Mailboxes between backends
There is currently no 100% foolproof way to do this, however, if you
issue a rename command to a frontend (as you would to move a mailbox
between partitions), and replace the partition name with the name of
the new backend, it will move the mailbox to the indicated backend. You
can also use the format backend.domain.com!partition to move to a
specific partition (otherwise the default partition will be used). In
cyradm, this looks like:
cyrus.andrew.cmu.edu> rename user.bcyrus user.bcyrus mail2.andrew.cmu.edu!u2
Note that since seen state is stored per-user, it is possible that when
moving a shared mailbox users will have strange effects. The general
rule is that moving an INBOX will move the entire user (including all
sub-mailboxes to the INBOX, and seen state, and subscriptions, and
sieve scripts, etc). The seen state is merged with the seen state on
the new backend, so that no data is lost (seen state is also the only
part left behind on the source backend). In the case of any other
mailbox, however, only that individual mailbox is moved. If it is a
quota root, the new quota root is instated on the new server, but
otherwise quotas can appear to be violated, since each backend only
takes care of its own quota.
In general, its better to leave trees of mailboxes on the same server,
and not move submailboxes of inboxes between servers.
Adding additional backend servers
This is very easy to do, simply configure an empty backend server and
set its mupdate_server parameter to point at the mupdate master. Then,
issue mailbox creates to it as you would any other backend server.
Backups
xxx, need to write stuff. You don't need to really backup the data on
the mupdate master or slaves, since this data can all be generated
directly from the backends quite easily.
Gotchyas
* Clients dealing with a pool of frontend servers - Some clients may
not be terribly efficient caching connections to a pool of imap
servers, this isn't a problem, per se, but it may mean that you
will see many more authentications than you are used to.
* Kerberos issues - If you are using kerberos authentication, you
will want to ensure that all your machines are keyed properly,
as we have seen problems with different clients trying to
authenticate to different services (e.g. imap.imap-pool instead
of imap.pool-frontend-1), so test the clients in use in your
enviornment and be sure that they work with whatever keying
scheme you use.
* Clients dealing with referrals - Some clients (we've had particuar
trouble with pine, though most of these issues have now been
resolved and new versions should be OK (that is, pine > 4.44), but
as referrals have not been extensively used by any IMAP server
until now, referrals are very likely to not work correctly or have
surprising effects.
* Clients dealing with getting a NO on LSUB commands - Some clients
(Outlook, for example) may behave poorly if an LSUB command returns
a NO, which may be the case if the backend server with the user's
inbox is down. We have, for example, seen this result in the
deletion of the disconnected message cache.
* Behavior of cyradm / some mailbox operations - The behavior of some
administrative commands might be slightly unexpected. For example,
you can only issue a SETQUOTA to a frontend server if the entire
mailbox tree underneath where you are setting the quota exists on
the same backend server, otherwise you will need to connect
directly to the backend servers to perform the needed changes.
Similarally, mailboxes will be created on the same backend server
that their parent is in. In order to create them on a different
server (or to create a new top level mailbox) you will need to
connect directly to the desired backend server.
* Subscriptions - If users want subscribe to a mailbox other than on
their backend home server, they won't be able to, unless you set
"allowallsubscribe: t" in the backend imapd.confs. This essentially
lets any string be subscribed to successfully.
* Restarting the mupdate master - Because ctl_cyrusdb -r clears
reservations on mailbox, if you restart the mupdate master (and run
recovery), then this could (we suspect, very rarely) lead to
inconsistancies in the mupdate database.
Troubleshooting & when things go wrong
* Mailbox operations are being denied - This is an indication that
the mupdate master may be down. Restart it.
* Mailbox operations are not being seen by one or more frontends -
This indicates that the mupdate process on a slave may have died,
you may need to restart master. Alternatively, mupdate will retry
connections every 20 seconds or so for about 20 attempts if the
master does go down.
* A frontend's mailboxes.db is corrupt / out of sync - Restart master
on the frontend, and have the mupdate process resynch the local
database. You may need to remove the local mailboxes database if
the corruption is extreme.
* A mailbox's location keeps switching between two (or more) backend
hosts. - It probably actually exists on both hosts. Delete the
mailbox from all but one of the hosts, and run a ctl_mboxlist -m on
the one where you want it to actually live.
* Databases are never created on the frontends/slaves - Check to
ensure that the mupdate slave process is started, (is prefork=1)
References
* Aggregator Overview
* MUPDATE Protocol Reference