Large enterprises must store a lot of data, and it only continues to grow. Given the risk of deleting data to make more space on storage hardware, the need to archive and continuously protect almost infinite amounts of data has inspired many to look for answers in software-defined object storage. The Linux Foundation recently announced a new project to advance cloud object storage technology. A Techcrunch piece on the Kinetic project highlights why data centers the world over are rushing to migrate away from old-school file-based systems, to get to object storage. The data mushrooming today is increasingly unstructured and unchanging: images, videos, transactions. Such data is best accessed and managed directly by a unique key, not a file path, so that the underlying disk hardware can morph and scale to meet the ever-growing demand.
The Kinetic Open Source Storage project goal is “to define and promote open source software and standards for cloud object storage technologies.” This will free hardware vendors from having to develop their own proprietary software for such storage on their products. Driving this new cooperation across the major disk manufacturers are not only the cost-benefits of shared development, but also the growing demand for open, general solutions that reduce the lock-in and support risks. Kinetic should meet basic requirements for entry-level object storage; however, there are several limitations that will be problematic for large-scale enterprises. The bigger the enterprise, the more likely it is to need a pure object storage, like Caringo Swarm. Limitations that must be considered include:
- Too much traffic. Enterprise software needs a fast way to locate objects without involving all of the drives and spamming the network. Complete object storage solutions, such as Swarm, offer some kind of dynamic master index that spans all objects in the cluster so that there’s no need to blast requests cluster-wide. When software requests an object from a Swarm storage cluster, the RAM-based Overlay Index locates the nodes containing replicas of the object and directs the request to the best node, without multicast noise.
- Too slow for very large objects. The size limit on objects hurts performance for storage of very large objects. Enterprise object storage solutions need to provide ample performance and additional support for very large objects, both in how quickly they can be written and in how efficiently they can be stored. With Swarm, software can take advantage of parallel write (which divides an object into multiple parts and uploads them simultaneously) and erasure coding (which segments and stores large objects with great efficiency and security).
- Too little hardware flexibility. Complete object storage solutions, such as Swarm, let enterprises blend several classes of storage in the same storage cluster, including SMR and flash. The problem with Smart drives is that they restrict object representation to a form that is convenient for one class of device only.
Cooperative efforts in the industry are to be commended, and they often solve issues for a number of organizations. However, for meeting the demands of large enterprises that need to be able to use a mix of commodity hardware to provide secure storage within their own data center or in a storage cloud—and who need almost infinite scalability—a proven, feature-rich object storage software solution is more likely to be the right answer. No object storage software vendor has been in market longer than Caringo. We’ve been working for a decade to solve the biggest storage and compliance challenges large enterprises face with Swarm—simple, bulletproof, limitless object storage.
For more information on how we have been solving large scale storage challenges (and how we can do the same for you) view our recent Object Storage Primer webinar or contact us to setup an overview session.
While object storage isn’t a panacea, it is an increasingly important storage technology that enables on-demand access for video workflows. More Details »