Best Practices with Backup and Archival

With every mail transaction, information is being added to the enterprise mail store. This information could be useful as a reference for business purposes or in possible future legal investigations. In many countries now, retaining this information has become mandatory by law. Besides this, in order to optimize the usage of the space available on the mail store, a copy of the old messages is made and stored away in an offline storage device. This ensures that the system efficiency does not degrade due to bloating of the mail store.

Customer would like to ensure security of the mail accessed by users to guard against PC crashes, or laptop losses. The objective being that the organisation to have a copy of every mail sent or received, for a defined period of time, by selected or all users irrespective of how they access their mail. The customer would also like to provide the end user the capability to retrieve his own mail from the archive store.

An archive differs from a backup as follows:

  • A backup contains the snap shot of the mail store. Therefore a backup will not contain mail which have been deleted from the mailbox or downloaded to the users PC. Whereas an archive contains all the mail sent and received by the user.
  • Archive is arranged day wise whereas as mentioned earlier backup is a snap shot
  • Archiving happens in 'real time' where as backup is a scheduled activity.

Best Practices

Daily system/domain backup to a remote machine

To help rebuild the system in case of a crash, we need to take a system and domain data backup for each server. The jobs for this can be scheduled. The backups are to be taken offline using a third party backup tool. Please note that each server should have its own separate backup job since the server properties would differ on each.


Regular and incremental additive backups of the mailstore

The live mailstore is stored in a format called “Maildir”. This format is a simple folder/file structure on the ext3 file system of the Linux server. One folder, one domain, within which there are folders for each user, within which there are personal folders and mail.

Configure the integrated or third party backup solution (like net backup, or veritas etc.) to ensure that an offline copy of the mailstore is available to allow restoration upto a particular date or just a few mail files or a folder etc. Using this, ensure that the mailstore on the server is properly secured to another medium. This is suitable for users, whose mails reside on the server e.g. IMAP users, Web client users, POP users (who leave a copy on the server).

You could configure this to backup in an additive mode i.e. it wont delete mail from the backup even though the user may have deleted from his mail folders.

The restoration here from the third party backup tools can be for a single user or even a single folder or mail since that is the granularity of storage. This solution can be used to restore a part or the complete mailbox of a user.

This can however miss messages which come and are deleted between the two backup schedules (covered by the archive solution).


Daily system/domain backup to a remote machine

To help rebuild the system in case of a crash, we need to take a system and domain data backup for each server. The jobs for this can be scheduled. The backups are to be taken offline using a third party backup tool. Please note that each server should have its own separate backup job since the server properties would differ on each.


Regular and incremental additive backups of the mailstore

The live mailstore is stored in a format called “Maildir”. This format is a simple folder/file structure on the ext3 file system of the Linux server. One folder, one domain, within which there are folders for each user, within which there are personal folders and mail.

Configure the integrated or third party backup solution (like net backup, or veritas etc.) to ensure that an offline copy of the mailstore is available to allow restoration upto a particular date or just a few mail files or a folder etc. Using this, ensure that the mailstore on the server is properly secured to another medium. This is suitable for users, whose mails reside on the server e.g. IMAP users, Web client users, POP users (who leave a copy on the server).

You could configure this to backup in an additive mode i.e. it wont delete mail from the backup even though the user may have deleted from his mail folders.

The restoration here from the third party backup tools can be for a single user or even a single folder or mail since that is the granularity of storage. This solution can be used to restore a part or the complete mailbox of a user.

This can however miss messages which come and are deleted between the two backup schedules (covered by the archive solution).


Daily snapshot of the mailstore

Configure a third party solution to take a snapshot of the mailstore once or twice a day. In case of a crash, restore to the last good state and start. This can miss messages after the last snapshot


Leave copy for critical users for POP

For IMAP users (web client and IMAP users), the mail anyways reside only on the server. Users may of course delete mail, but the solution mentioned in point (1) above secures against this.This option may be enabled for POP users (specific only). Note that it will increase live storage requirements.


User level Personal Mail archival

This solution configures a 'tap' in the mail flow for selected or all users, and transports a copy of every mail sent and received to the archive server. The archive server has a corresponding user account to store these mail. E.g. Mail sent and received by user1@domain.com are archived into user1@archive.domain.com. This is a regular mailbox which can have mail retention policies of 6 months to 1 year as per policy. The end users can have the control to login to this account and retrieve mail by forwarding to the original account.

Each user can be mapped to one or more archive accounts. The mailboxes for the archive accounts can reside on the same server or on another server dedicated for the purposes. The basic requirement is that of additional dedicated storage (it's size depending on the number of users, mail transaction volume, and retention policies).

The archive accounts can be retained for a specified period. Messages that fall outside the specified period can either be deleted or compressed and moved to an alternate storage for longer term archival as per the business policy of the enterprise.

The personal archive store is also stored in the “Maildir” format and is organized similarly. The same principles of backup would apply to this as well. The restoration here from the third party backup tools can be for a single user or even a single folder or mail since that is the granularity of storage

Note: The mail in the archive store are all in the Inbox only. The personal mail folders created in the user's account are not reflected here. At the most a filter may be deployed to separate and differentiate sent items into the sent items folder.

In case of a loss of entire mail store of a user, its possible to restore the entire archive store of that user to the live mailstore. This is done using a simple tar copy of the folder and restoring it on the live store. Please take note of the above Note. The user will certainly get all the mail for the period of retention in the archive, but will lose the personal folders created if any.


Mithi Connect Server offers several different configurations for archiving the messages, as explained below:

Configuration Options Usage Benefits
Single Archive Account Mail for all users saved to a single archive account E.g. user1@domain.com => archive@domain.com, user2@domain.com => archive@domain.com
  • Access to archive by administrator
  • Search & forward mail from the archive
  • Useful for smaller organizations
  • Easier to maintain with access only by administrator
Group Archive Account Mail for groups of users saved to respective group archive accounts. E.g. user1@domain.com => salesarchive@domain.com, user2@domain.com => salesarchive@domain.com, user3@domain.com => adminarchive@domain.com, user4@domain.com => adminarchive@domain.com
  • Access by respective group leaders/ administrators.
  • Search & forward mail from the archive.
  • Useful for medium to large organizations.
  • Segregated archive making it easier to manage by separate people.
Individual Archive Account Mail for each user saved to a corresponding individual user archive account. E.g. user1@domain.com => user1@archive.domain.com, user2@domain.com => user2@archive.domain.com
  • Option to allow certain/all end users to access their own archive accounts to check and restore their mail.
  • Search and restore mail by forwarding or moving between two IMAP accounts.
  • Useful for larger organizations.
  • Enabling users to access their own archives makes it even easier to manage.

Compliance Mail Archival

This solution configures selected or all users to transport a copy of every mail sent or received through the server to the archive server, where the mail is indexed, and stored. On demand the administrator may access this archive store through a search interface, look for mail using various mail attributes like sender, recipient, content etc, and retrieve them. The storage for this is compressed and encrypted to secure against direct access from the backend.

This solution is primarily suited for audits and compliance and is being increasingly adopted by various organisations. Strongly recommended to deploy this.

This solution cannot be used to restore a user's mailbox or a part thereof.

The compliance archive store is also stored in a folder structure with files which are compressed and encrypted. The backups can be taken using the same principles, but the restoration has to happen for the entire store.

The compression works on two basic principles, viz. De-duplication of the mail and also compression of the mail, including the attachments.

We feel an appropriate combination of the above 6 approaches will provide the required security and peace-of-mind.


Use Cases

Server crash

Rebuild the server by installing OS, Connect XF and restoring the backup.


User loses laptop or PC crashes

Restore his mailbox from the backup upto a particular state, and let him redownload all his mail. He would get extra mail.


Investigation request for a particular mail

Look in the compliance archive to investigate.


User complains of a loss of a few messages.

Encourage user to go to the user level archive and retrieve it themselves


User refuting that he/she ever got a mail

Confirm from the compliance Archive.


FAQs

If an email is sent to multiple recipients and with attachments, what is the strategy for storage in both the archival systems to save on storage space?

In the personal archive store, the mail will be duplicated into each recipient's mailbox and will add to the storage requirement.

However for the compliance archive store, the mail is de-duped and also compressed to save on storage space. De-duplication means a single physical copy with internal links.