It used to be that the world of High-Performance Computing (HPC) was relegated to large research organizations. That is no longer the case. The need for HPC has escalated along with the explosion of data sets (what some might refer to as “big data”) and the sophistication of today’s business organizations. No longer is simple transactional computing adequate for the enterprise. Today’s savvy business executives want to be able to crunch numbers and analyze data stores to gain insight that drives opportunities and profit margins.
What are the 3Vs of Big Data?
The 3Vs (introduced by Gartner analyst Doug Laney in 2001) of Big Data are volume, variety and velocity. According to Laney, the challenges of big data management result not just from the volume alone but from the combination of those characteristics.
Let’s break these down one by one and talk about the impact of using object storage on each of the “Vs.”
Volume: How Does Object Storage Manage Large Stores of Data?
Scalability varies with different products, but a best-of-breed object storage technology will be highly scalable. Caringo Swarm Object Storage scales to hundreds of petabytes and beyond. It should also include robust cluster and data management features. Another desirable characteristic is that the Swarm cluster actually becomes more efficient as it grows. This efficiency was designed into Caringo Object Storage from the beginning and continues to differentiate our solution from others in the marketplace. We have customers running extremely large Swarm clusters in the field that have been able to take advantage of our “scale-up, scale-out” approach for over a decade.
Swarm Content Portal simplifies management of large stores of data by making it easy to use custom metadata tags and use policies to define data retention and tiering.
Variety: Can Object Storage Handle Different Data Types?
While object storage is often touted as a solution for unstructured data, it actually can be used for many types of data. Thanks to broad use of Amazon S3 and lightweight protocol converters (such as SwarmNFS), object data becomes equally accessible via S3, NFS, or any custom application you have integrated with Swarm. All of this while referencing the same name space!
Velocity: Is Object Storage Too Slow to Be Useful?
Historically, object storage has been thought of as slow; however, that is not the case with Swarm Object Storage. In recent benchmark tests at the UK’s Science and Technology Facilities Council’s Super Data Cluster Environment, Swarm Object Storage was tested along with the technology of other vendors. Swarm Object Storage bested the competition and achieved read performance characteristics similar to existing parallel file systems.
Where Can I Learn More About Object Storage for HPC?
On October 22, our Tech Tuesday educational webinar will be Object Storage Solutions for High-Performance Computing. Join me and Adrian “AJ” Herrera live at 11am PT/2pm ET (and bring your questions) or watch on demand after the event.
You can also email us if you have questions set up a personalized demonstration with one of our Storage Engineers.
