Storage efficiency

Storage efficiency is the ability to store and manage data that consumes the least amount of space with little to no impact on performance; resulting in a lower total operational cost. Efficiency addresses the real-world demands of managing costs, reducing complexity and limiting risk. The Storage Networking Industry Association (SNIA) defines storage efficiency in the SNIA Dictionary as follows:

[math]\displaystyle{ \text{storage efficiency} = \frac{\text{effective capacity} + \text{free capacity}}{\text{raw capacity}}. }[/math]

The efficiency of an empty enterprise level system is commonly in the 40–70% range, depending on what combination of RAID, mirroring and other data protection technologies are deployed, and may be even lower for highly redundant remotely mirrored systems. As data is stored on the system, technologies such as deduplication and compression may store data at a greater than 1-to-1 data size-to-space consumed ratio, and efficiency rises, often to over 100% for primary data, and thousands of percent for backup data.

Technologies

Different technologies exist at different and sometimes multiple levels:

Snapshot technology—known formally as "delta snapshot technology"—gives the ability to use the same dataset multiple times for multiple reasons, while storing only the changes between each dataset. Some storage vendors integrate their snapshot capabilities at the operating system and/or application level, enabling access to the data the snapshots are holding at the system and/or application management layers. Terminology around snapshots and "clones" is currently confusing, and care must be taken when evaluating vendor claims. In particular, some vendors call full point-in-time copies "snapshots" or "clones", while others use the same terms to refer to shared-block "delta" snapshots or clones. And some implementations can only do read-only snapshots, while others are able to provide writable ones as well.

Data deduplication technology can be used to very efficiently track and remove duplicate blocks of data inside a storage unit. There are a multitude of implementations, each with their separate advantages and disadvantages. Deduplication is most efficient at the shared storage layer, however, implementations in software and even databases exist. The most suitable candidates for deduplication are backup and platform virtualization, because both applications typically produce or use a lot of almost identical copies. However, some vendors are now offering in-place deduplication, which deduplicates primary storage.

Thin provisioning technology is a technique to prevent under-utilization by sharing the allocated, but not yet utilized capacity. A good example is Gmail, where every Gmail account has a large amount of allocated capacity. Because most Gmail users only use a fraction of the allocated capacity, this "free space" is "shared" among all Gmail users.

Major advantages

Actively increasing storage efficiency using these techniques has the following advantages:

Backup and restore. Using snapshots, time used for both backup and restore RTO can be minimized. This can greatly reduce cost, and reduce hours of downtime to seconds of downtime. Snapshots also allow for better RPO values.

Reducing floorspace. When less storage is required to store a given amount of data, less data center floorspace is required.

Reducing energy use. When fewer spindles are required to store a given amount of data, less power is required.

Provisioning efficiency. Writable delta snapshot technology allows for very fast provisioning of writable data copies. This reduces waiting time in processes that require that data. Examples are data mining, test data, etc. Snapshot integration at the OS and/or application level also leads to faster provisioning, because system and/or application managers are able to manage their own snapshots without having to wait for storage managers and/or provisioning procedures.

Major commercial players

All major vendors are implementing one or more of these technologies, because storage efficiency is becoming more and more popular. Customers are facing storage requirements that are growing exponentially and a strong demand for cost-cutting. The major vendors are NetApp, EMC, HDS, IBM and HP.