Data Integrity in Cloud Storage

Cloud storage is certainly a very important part of an overall cloud computing strategy. Data, after all, is at the heart of all clouds, whether public or private. Further, all attributes that make up a robust cloud offering - elasticity, availability and security in particular - apply to the storage elements within a cloud as well as the compute elements.

One area that is often overlooked in cloud storage is data integrity. Too often, we assume that storage arrays receive, store and retrieve data flawlessly. Studies have shown otherwise - in fact, the phenomenon of silent data corruption is real. A landmark paper was published by CERN in 2007 that explored the problem in great detail. The various well-known RAID techniques do nothing to prevent the problem.

To mitigate against the loss of integrity in private and public clouds, one must insure that their storage is protected by the new Data Integrity Field standard, called 'DIF' for short. DIF can provide either storage-centric or end-to-end data integrity. It insures not only the data itself is protected, but that the addresses of the I/O are correct - after all, you don't want good data at the wrong address in your cloud storage.

The bottom line on DIF is this - when using cloud storage, either public or private, best practice is to store cloud data on disks or arrays that perform the DIF function. Consumers of cloud storage should ask their provider about DIF. It is a further best practice to use cloud applications with DIF built-in, to provide end-to-end protection.

I think cloud storage is a fascinating subject with many aspects that businesses must consider to fully take advantage of it - and I am happy to blog here on cloudslam.com about cloud storage. Let me know your thoughts on cloud storage, and stay abreast of the emerging standards such as the SNIA Cloud Data Management Interface (CDMI).

Happy Holidays to all!