Transforming Enterprise Data Storage for HPC

The need for HPC in Enterprise Data Storage has escalated with the explosion of big data. Learn how Object Storage can help.

HPC-Banner
HPC-Banner

It used to be that the world of High-Performance Computing (HPC) was relegated to large research organizations. That is no longer the case. The need for HPC has escalated along with the explosion of data sets (what some might refer to as “big data”) and the sophistication of today’s business organizations. No longer is simple transactional computing adequate for the enterprise. Today’s savvy business executives want to be able to crunch numbers and analyze data stores to gain insight that drives opportunities and profit margins.

What are the 3Vs of Big Data?

  1. Volume
  2. Variety
  3. Velocity

The 3Vs (introduced by Gartner analyst Doug Laney in 2001) of Big Data are volume, variety and velocity. According to Laney, the challenges of big data management result not just from the volume alone but from the combination of those characteristics.

Let’s break these down one by one and talk about the impact of using object storage on each of the “Vs.”

Volume: How Does Object Storage Manage Large Stores of Data?

Scalability varies with different products, but a best-of-breed object storage technology will be highly scalable. Caringo Swarm Object Storage scales to hundreds of petabytes and beyond. It should also include robust cluster and data management features. Another desirable characteristic is that the Swarm cluster actually becomes more efficient as it grows. This efficiency was designed into Caringo Object Storage from the beginning and continues to differentiate our solution from others in the marketplace. We have customers running extremely large Swarm clusters in the field that have been able to take advantage of our “scale-up, scale-out” approach for over a decade.

Swarm Content Portal simplifies management of large stores of data by making it easy to use custom metadata tags and use policies to define data retention and tiering.

Variety: Can Object Storage Handle Different Data Types?

While object storage is often touted as a solution for unstructured data, it actually can be used for many types of data. Thanks to broad use of Amazon S3 and lightweight protocol converters (such as SwarmNFS), object data becomes equally accessible via S3, NFS, or any custom application you have integrated with Swarm. All of this while referencing the same name space!

Velocity: Is Object Storage Too Slow to Be Useful?

Historically, object storage has been thought of as slow; however, that is not the case with Swarm Object Storage. In recent benchmark tests at the UK’s Science and Technology Facilities Council’s Super Data Cluster Environment, Swarm Object Storage was tested along with the technology of other vendors. Swarm Object Storage bested the competition and achieved read performance characteristics similar to existing parallel file systems.

Caringo Swarm object storage for HPC data sets

Where Can I Learn More About Object Storage for HPC?

On October 22, our Tech Tuesday educational webinar will be Object Storage Solutions for High-Performance Computing. Join me and Adrian “AJ” Herrera live at 11am PT/2pm ET (and bring your questions) or watch on demand after the event.

Register Now

You can also email us if you have questions set up a personalized demonstration with one of our Storage Engineers.

John Bell
John Bell

About The Author

John Bell has been with Caringo for 7 years and is our Senior Consultant on the design, implementation and management of object storage. John has worked with many of our largest customers to transform their storage infrastructure with highly efficient and cost-effective object storage solutions. John has a B.S. in Aeronautical Engineering from the University of Illinois at Urbana-Champaign and a B.S. in Computer Science from Indiana State University.


More From Author