Strategies to manage the rapidly growing
storage needs for your enterprise

Over the last year or so, we have been experiencing a rapid growth in storage needs especially with regards to the mail service for both our customers and ourselves.

Managing this continuous growth for storage is a fine balance between
a. Allowing the flexibility to use as much storage without being distracted about managing the overflow
AND
b. Controlling the bloat to be able to manage the backup, recovery and restore processes. Several challenges confront an enterprise in managing its storage.

  • Frequent storage overflow.
  • Accidental deletion of information.
  • Audit, compliance and investigative requirements.
  • Failure or loss of any server, PC, laptop in the organisation requiring a full rebuild with latest, closest snapshot of the data.
  • Failure of any of the storage devices.
  • Failure of the entire site.

A simple 3 part storage strategy comprising of:

  • Choice of storage components/devices.
  • Defining and enforcing usage policies.
  • Securing the data for compliance and recovery, can help manage, recover from and respond to above challenges effectively.

This note outlines an approach to putting in a simple, workable storage management strategy for your IT infrastructure. Many of the ideas presented here are used by Mithi in house,while several have come from our interactions with customers.


Choosing the Components

Preventing system failure: The Starting point is about reducing the risk and chance of failure at the system.Irrespective of what devices you use in the enterprise, ensure that you:

  • Provide quality power supply to all equipment to reduce chances of burn out/damage.
  • Use carrier grade servers with redundant power supplies and RAID for storage reliability.
  • Use SAN and NAS depending on your IT infrastructure and what your budget may permit
  • Improve efficiency since this allows you to automate your corporate mail usage policies.

Storing data on the network: It is also quite natural for the storage & server resources to be distributed amongst offices and locations (stemming from the organic or unplanned growth in an organization)

In such a situation, it is possible that if shared storage devices are not used, the data would be splintered across all the devices. Companies (depending on the available budget) adopt shared storage devices like a SAN and NAS to physically consolidate all the data, while still providing a separate logical partition to the applications. This tends to greatly improve the reliability and simplify the management of the data (since these devices have built in redundancy and can be managed at a single point)

Cost effective file server: If your budgets are limited, you could opt for a simple server to act as shared file servers and encourage or enforce storage on the network file server in shared folders.


Defining and enforcing usage policies

  • Quota: Allocate a fixed limited resource to each user. It becomes the user's responsibility to clean this regularly. This also ensures that there can be no indiscriminate or lop sided usage of the resource by any member of the team.
  • Partioning
  • Compression: Store older, infrequently needed data by compressing it to reclaim storage space. E.g. in the email archival system, mail for each may be compressed daily.
  • Auto deletion policies: Configure automatic deletion of older data (as appropriate and as per company policy) to reclaim space.

Securing the data for compliance and recovery

Irrespective of the size of your organization, we recommend that you follow these principles while designing your backup strategy:

  • Have a map of all the data and applications (information assets) in the organization and where it all resides (servers, PCs, laptops, storage devices).
  • Using appropriate tools (refer table below), ensure that each information asset is being backed up periodically.
  • Store all the backups in a central, dedicated resource.
  • Put in policies and processes to rotate these backups (depending on your policies and how far back you might ever need to go).
  • Ensure that periodic offsite backups form part of the above processes.
  • Regularly review the map, the backups and also run mock restorations from the backups to ensure that the back-ups are in good condition.

Email archival system or Backups: Archival is about maintaining the data generated in the system and about being able track all possible transactions for a specified period of data, primarily for the purpose of compliance.

Archival is very difficult to achieve by using backups since in the space between backups, we could have transactions that can be reversed and we would not have a trace of those. Archival is ideally achieved if the application supports archival of the transactions AS THEY HAPPEN, transparent to the end user such that irrespective of the user actions, each action is recorded with the relevant information in a separate store. E.g. database transactional applications might record each transaction as an event and maintain events for the specified period, email systems could maintain a copy of each and every mail sent or received in a separate store.


What all do you need to backup:

Reference: Strategies

  • Email, as this is construed as official communication.
  • All the information from your financial software.
  • All your web pages, databases, and anything that you made or would have trouble replacing.
  • All the information from inventory control, customer databases, or other specialist business software.
  • Important correspondences.
  • Internal documents (important memos and the like).
  • Anything else, which if lost will negatively impact the business.

What you need not backup:

  • Operating System, as you would have installable disks.
  • Application software.
  • Temporary files (At Mithi users share documents using a file server over shared folder called "Temporary Data Exchange". This area is periodically deleted to reclaim the space. Without such a folder, users tend to use email, to exchange temporary "Work In Progress" information, leading to bloat in the email storage requirement.)

Recommended ways to backup your data from the various servers, PCs, laptops and shared storage

Organization
SizeParameter
How
(process/method)
When Where Tools
Small(10clients,50GB) Simple FULL backup Daily, Weekly, Monthly rotation External drives, Flash Drive, Offsite Copy, Rsync, Tar Windows or native platform backup tools, and application specific backup commands
Meduim(50clients,200GB) Full backup + Incremental backup Daily, Weekly, Monthly, Yearly rotation External drives, Tapes, PC dedicated for backups, Offsite Rsync, Third party tools which support incremental backups, Home grown tools/scripts, Application specific backup commands
Large(10clients,50GB) Continuous (real time) + Network (centralized for all resources) Immediate, Daily, Weekly, Monthly rotation Backup servers, Storage devices, Tapes Offsite backups Third party backup software like Veritas, Netvault, Application specific backup commands

Case Study of the storage and backup strategy in a small organization

The figure below depicts how a small organization, which has an infrastructure comprising of different types of servers, different platforms, different applications, local and remote servers, can put up a simple, reliable backup infrastructure. It is based on all the principles outlined in the section above. A similar framework can easily be adapted for the medium and large organizations with variations in the tools, periods, and devices used.

Storage and Backup Strategies in Small Organization

References:
Strategies

The above note is brief summary of possible storage strategies and not a comprehensive document on all possible storage strategies and technologies. Neither is it a document on Business Continuity Strategies.