<img class="size-medium wp-image-5588 alignright" src="https://i0.wp.com/www.caringo.com/wp-content/uploads/2016/05/Metadata-Storage-Solutions.png?resize=300%2C231&ssl=1" alt="Metadata Storage Solutions" width="300" height="231" srcset="https://i0.wp.com/www.caringo.com/wp-content/uploads/2016/05/Metadata-Storage-Solutions.png?resize=300%2C231&ssl=1 300w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2016/05/Metadata-Storage-Solutions.png?w=372&ssl=1 372w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" />A couple of weeks ago, I discussed <a href="https://www.caringo.com/blog/protecting-healthcare-data-raid-vs-advanced-erasure-coding/" target="_blank" rel="noopener noreferrer">RAID vs. advanced erasure coding</a> in the context of protecting vital healthcare data. This week, I want to talk about some issues around the storage of object metadata.
One of the great advantages of object storage is that each object can have a collection of metadata that describes the object. Metadata may include a name for the object, a location, the format or type of the data, categories that it is a member of, etc. Some object storage systems store this metadata in SQL databases or worse, flat files in their underlying filesystem’s directory, which are both terrible architectural models. Why?
For starters, the database (or flat files) become the bottleneck for the storage system because they are hit so often by requests from applications, resulting in the database or storage nodes becoming bogged down, severely impacting scalability. That’s why NoSQL is so popular in large-scale or big data applications—because it scales so much better than traditional SQL or flat file approaches.
As we all know, SQL databases require ongoing administration and the schema are rigid, stifling the ability of an organization to extend metadata with additional information that are relevant to the application. In the case of using flat files instead of a real database, additional problems crop up such as rolling back a transaction, recovering from a data corruption event or getting a reasonable-sized answer set for a given query.
Lastly, and this is the one that makes me tremble in fear, what if a failure occurs? What if the storage of the objects somehow becomes out of sync with the metadata database because a database server or a switch or an application or something else failed or hiccuped? When the metadata database has to be updated every time an object is updated, it is the responsibility of the application to keep the metadata database in sync with the objects that are being stored. But what if the application gets confused or overwhelmed or crashes? And, honestly, do I have to administer the metadata database myself? Do I need to architect a proven, mission-critical metadata database infrastructure that can withstand drive failures or other kinds of failures and heal itself? Yikes!
Not to worry—as long as you are using Caringo storage systems. <a href="https://caringo.wistia.com/medias/txy745cn82" target="_blank" rel="noopener noreferrer">Swarm</a> stores the metadata with the objects so it can never be torn apart or get out of sync. The metadata is easily extended to incorporate additional custom elements and it is automatically protected by the <a href="https://www.caringo.com/resources/faqs.html#q9" target="_blank" rel="noopener noreferrer">elastic content protection</a> that is used in Caringo Swarm to protect the objects. Applications are not themselves responsible for keeping metadata in sync with the objects and they can ask any of the nodes in Swarm for an object’s metadata, and get just the answer set they need, so there is no bottleneck on a centralized SQL database server. Metadata storage, like every aspect of Caringo, is simple, bulletproof and limitless.
Have questions? Feel free to <a href="https://www.caringo.com/company/contact.html" target="_blank" rel="noopener noreferrer">contact us</a>, or join one of our upcoming webinars:
<ul>
<li>May 5: <a href="https://www.brighttalk.com/webcast/13173/198123" target="_blank" rel="noopener noreferrer">Radically Simplified Content Storage & Distribution for Social Discovery Sites</a></li>
<li>May 12: <a href="https://www.brighttalk.com/webcast/5583/204219" target="_blank" rel="noopener noreferrer">Designing Storage and Apps to Enable Data Monetization</a></li>
<li>May 26: <a href="https://www.brighttalk.com/webcast/13173/198121" target="_blank" rel="noopener noreferrer">Protecting Healthcare & Life Sciences Data</a></li>
</ul>

The storage and search of metadata has just become easier with object storage. Never store your metadata in SQL databases or flat files again.

Don studied electrical engineering at Cornell University followed by a career in high-tech that included five years at Intel, ten years at HP, seven years at Microsoft and five years at EMC—to name a few. Don’s area of expertise is product management and product marketing for deep high-tech products.

Don Chouinard

The rise of big data brings extremely large data sets. Caringo Swarm Object Storage stores that data at up to 75% less TCO than traditional storage.

Big Data

Swarm object storage provides continuous, built-in data resilience to ensure data durability using redundancy to give you constant data availability.

Data Durability

Data integrity (i.e., maintenance of accurate data) is a critical function of storage systems. Caringo Swarm uses Content-MD5, Integrity Seals & WORM.

Data Integrity

With Caringo Swarm Object Storage, you get continuous, built-in data protection along with exceptional resilience, recovery, and content integrity. 

Data Protection

Elasticsearch is a distributed, multitenant-capable full-text search engine with HTTP web interface; it is fully integrated into Swarm Object Storage.

Elasticsearch

Metadata is "data about data" and can be descriptive, structural, administrative, reference or statistical metadata that is auto- or user-generated.

Metadata

A NoSQL database provides a mechanism for storage and retrieval of data other than the tabular relations used in traditional relational databases.

NoSQL

Object storage (object-based storage) is computer data storage architecture that manages data as objects, as opposed to file storage & block storage.

Object Storage

Metadata Storage Solutions

<a href="https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/10/Nerd-Center.jpg?ssl=1"><img class="wp-image-2504 alignright" src="https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/10/Nerd-Center.jpg?resize=371%2C244&ssl=1" alt="Nerd Center" width="371" height="244" srcset="https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/10/Nerd-Center.jpg?zoom=2&resize=371%2C244&ssl=1 742w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/10/Nerd-Center.jpg?zoom=3&resize=371%2C244&ssl=1 1113w" sizes="(max-width: 371px) 100vw, 371px" data-recalc-dims="1" /></a>This week, I’m prepping to present at tomorrow night’s Virtualization Meetup Group (VMUG) in Boston at the Microsoft N.E.R.D Center. Yes, there is still time to register! We will be presenting “A Crash Course on Object Storage. How It’s Helping Businesses in Every Industry and How It Can Help You!” at 8:00 p.m. With up to 90% of data inactive in some environments and average file sizes growing, organizations are looking beyond traditional storage for answers. According to IDG, unstructured data is growing at the rate of 62% per year and by 2022, 93% of all data in the digital universe will be unstructured. (Check out this blog on why “unstructured data is sexy” on the Headwaters Group news page.) CPU and Network technology are significantly ahead of storage performance and compliance continues to be hot button area (in the top 5 per CIO. com). We know that cloud adoption and integration are high priorities for many organizations as is increased demand for availability from more types of devices. Often, that leaves less budget to dedicate to physical storage and infrastructure. Are you being asked to store petabytes of data, make it searchable, accessible by any enterprise/cloud application or device, and guarantee integrity for decades—all with the same or maybe a modest increase in budget? Guess what? You aren’t alone. In our session, we will share our experiences working with hundreds of customers of various sizes across a number of different use cases with a focus on: Where object storage sits in the rapidly evolving storage market Trending use cases in the Medical Imaging, Service Provider, Web Content Delivery, Evidence Storage & Open Data Center markets The common issues object storage solves across these use cases How you determine if object storage can solve your data management issues I shared my professional history with you this summer prior to VTUG Summer Slam. Let me tell you just a bit about my colleague and co-presenter, Nilesh Kunvarji. Nilesh has 20+ years of experience in IT with 15+ years providing Enterprise Storage Management Solutions for SAN, NAS, backup/recovery, DR solutions, email management, and availability solutions to SMB and large corporate customers and strategic partners. Nilesh joined the Caringo team a bit over a year ago and is our East Coast technical lead. Most importantly, he’s the guy I call in to help with technical solutioning for our customers and potential customers. We’d love to meet you tomorrow night if you are at VMUG, or set up time to talk with you if you would like to brainstorm solutions for your storage needs! You can email us at info@caringo.com and follow us on LinkedIn, Twitter, Google+ and Facebook to keep up with our latest news.” target=”_blank” rel=”noopener”>Microsoft N.E.R.D Center</a>. Yes, there is still time to <a href="http://www.meetup.com/VirtGBoston/events/225571470/" target="_blank" rel="noopener noreferrer">register</a>! I will be presenting “A Crash Course on Object Storage. How It’s Helping Businesses in Every Industry and How It Can Help You!” at 8:00 p.m.
The main challenge with unstructured information is that it is difficult to determine what has value. With up to 90% of data inactive in some environments and average file sizes growing, organizations are looking beyond traditional storage for answers. According to IDG, unstructured data is growing at the rate of 62% per year and by 2022, 93% of all data in the digital universe will be unstructured. (Check out this <a href="http://www.theheadwatersgroup.com/your-unstructured-data-is-sexy/" target="_blank" rel="noopener noreferrer">blog</a> on why “unstructured data is sexy” on the Headwaters Group news page.) We know that cloud adoption and integration are high priorities for many organizations as is increased demand for availability from more types of devices. Often, that leaves less budget to dedicate to physical storage and infrastructure.
Are you being asked to store petabytes of data, make it searchable, accessible by any enterprise/cloud application or device, and guarantee integrity for decades—all with the same or maybe a modest increase in budget? You aren’t alone. In our session, we will share our experiences working with hundreds of customers of various sizes across a number of different use cases with a focus on:
<ul>
<li>Where object storage sits in the rapidly evolving storage market</li>
<li>Trending use cases in the Medical Imaging, Service Provider, Web Content Delivery, Evidence Storage & Open Data Center markets</li>
<li>The common issues object storage solves across these use cases</li>
<li>How you determine if object storage can solve your data management issues</li>
</ul>
I would love to meet you tomorrow night if you are at Virt-G, or set up time to talk with you if you would like to brainstorm solutions for your storage needs! You can email us at info@caringo.com and follow us on <a href="https://www.linkedin.com/company/caringo-inc-" target="_blank" rel="noopener noreferrer">LinkedIn</a>, <a href="https://twitter.com/CaringoStorage" target="_blank" rel="noopener noreferrer">Twitter</a>, <a href="https://plus.google.com/+Caringo/posts" target="_blank" rel="noopener noreferrer">Google+</a> and <a href="https://www.facebook.com/CaringoStorage" target="_blank" rel="noopener noreferrer">Facebook</a> to keep up with our latest news.

Ben Canter joined Caringo in 2015, and is now Caringo’s VP of Sales. Boston born and bred, Ben has spent 20+ years in IT, 13 on the VAR side, 4 years with manufacturing, 8 years focused on Data Center technology, and 5 years in Software-Defined Storage. Ben is certified in multiple technologies including HP, IBM, Dell, VMware, Cisco, and DataCore Software and thrives on helping customers solve problems.

Ben Canter

Caringo Swarm Object Storage provides an active archive (limitless & seamless pool of data) with data protection, management and search capabilities.

Cloud Architecture

Object-based storage (e.g., Amazon S3-compliant Caringo Swarm Object Storage) is the basis for all Cloud Storage⏤private, public or hybrid).

Cloud Storage

Caringo Swarm Object Storage uses both replication and erasure coding for flexible choice and automatic, adaptive transition between protection forms.

Erasure Coding

Software-defined storage (SDS) is data storage software for policy-based provisioning and management of data, independent of underlying hardware. 

Software Defined Storage

As you define your storage strategy, what are the required characteristics for each tier of your storage and what variables affect each storage tier?

Strategy

Crash Course on Object Storage at Virt-G Boston

<img src="https://i0.wp.com/www.caringo.com/wp-content/uploads/2019/07/tech-tuesday-data-resilience-recovery-swarm-object-storage.png?resize=480%2C270&ssl=1" alt="Data Resilience & Recovery in Swarm Object Storage" width="480" height="270" class="alignleft size-medium wp-image-14609" srcset="https://i0.wp.com/www.caringo.com/wp-content/uploads/2019/07/tech-tuesday-data-resilience-recovery-swarm-object-storage.png?resize=480%2C270&ssl=1 480w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2019/07/tech-tuesday-data-resilience-recovery-swarm-object-storage.png?resize=320%2C180&ssl=1 320w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2019/07/tech-tuesday-data-resilience-recovery-swarm-object-storage.png?w=640&ssl=1 640w" sizes="(max-width: 480px) 100vw, 480px" data-recalc-dims="1" />When it comes to protecting data, object storage provides significant advantages. In this educational webinar, we will discuss how Swarm Object Storage provides:
<ul>
<li>Continuous, built-in data resilience that can be configured to withstand massive hardware failure</li>
<li>Constant availability of data while maintaining 95% storage utilization for content</li>
<li>Automated data recovery eliminating the need for error-prone and time-consuming manual management</li>
</ul>

Learn how object storage can provide continuous, built-in data resilience that can be configured to withstand massive hardware failure

Zach Gover

Swarm Storage provides a complete S3-compliant platform for data protection, management, organization & search at massive scale for unstructured data.

Swarm Storage

Caringo Tech Tuesday webinars offer insight into Object Storage technology and how Swarm works, plus gives you access to our storage SMEs.

Tech Tuesday

Data Resilience & Recovery in Swarm Object Storage

<img class="float-left pad-right pad-bottom" src="https://i0.wp.com/www.caringo.com/wp-content/uploads/2016/08/Whitepaper_Replication_Erasure_Coding_Thumbnail.png?w=150&ssl=1" alt="Replication and Eradure Coding Explained" data-recalc-dims="1">Unlike what most storage vendors will try to make you believe, data loss in storage systems is unavoidable. It is right up there with death and taxes. However, by selecting the right solutions and applying the appropriate parameters, we can control to a large extent both the probability of data loss happening, as well as the size of the data loss when it actually does happen. This paper provides a qualitative, straightforward view into the specifics of this process as related to the Caringo object storage software, Swarm.

Unlike what most storage vendors will try to make you believe, data loss in storage systems is unavoidable.

Elastic Content Protection Technical Overview

<a href="https://www.caringo.com/dataprotection-whitepaper-thumbnail"><picture class="alignleft size-medium wp-image-12851 sp-no-webp"><source srcset="https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.webp 232w,https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.webp 500w" sizes="(max-width: 232px) 100vw, 232px" type="image/webp"><source srcset="https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.png?resize=232%2C300&ssl=1 232w, https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.png?w=500&ssl=1 500w" sizes="(max-width: 232px) 100vw, 232px"><img src="https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.png?resize=232%2C300&ssl=1" class="alignleft size-medium wp-image-12851 sp-no-webp" alt="Protecting Data with Caringo Swarm Object Storage" height="300" width="232" srcset="https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.png?resize=232%2C300&ssl=1 232w, https://i1.wp.com/www.caringo.com/wp-content/uploads/2018/11/DataProtection-Whitepaper-Thumbnail.png?w=500&ssl=1 500w" sizes="(max-width: 232px) 100vw, 232px" data-recalc-dims="1"></picture></a>With the onslaught of data that organizations are experiencing in the cloud age, protecting all that data has never been more important. Storing and providing access to your stored data may be the most visible thing that Swarm does, but Swarm also protects your data. Our founder, Jonathan Ring, uses a metaphor for Swarm as a ship carrying and protecting your data as the river of hardware changes over time, either through upgrade or eventual failure.

With the onslaught of data that organizations are experiencing in the cloud age, protecting all that data has never been more important.

Protecting Data with Caringo Swarm Object Storage

I caught the home improvement bug right in the middle of preparing for SC15…so while I’m down at the Austin Convention Center this week talking about Caringo’s Object Storage Software, my house will be getting new hardwood floors (tile was the past two weeks). So, why did I go with tile and hardwood? Simple…I never want to have to worry about my floors again. I want flooring that is virtually bulletproof. And now that I’ve ripped out a bathtub to put in a walk-in shower, I’ve realized that there is limitless potential in my house. Are these traits you want in your data storage?
These are the same three words that we use to describe Caringo Swarm—simple, bulletproof, limitless. The Swarm platform is ideally suited for high-performance computing (HPC) and Big Data since it was designed with massive scale in mind. Able to store trillions of objects of any size without degradation in performance with continuous, flexible data protection, Swarm gives you the same kind of durability that I’m going to get from my new floors.
We all have different learning styles, and I like to see things in action and then be able to ask questions (the tech writer in me I guess). So, at SC15, I want you to see our products in action and be able to get all your questions about object storage answered. In addition to the engineers that work here at our Austin headquarters, we’ve flown in a number of our key experts. We will have demos for you so you can get up close and personal with our products, including:
<ul>
<li><a href="https://www.caringo.com/products/swarm.html" target="_blank">Caringo Swarm</a>—Caringo’s software-defined object storage leverages simple and emergent behavior with decentralized coordination to handle any rate, flow or size of data. Swarm turns standard hardware into a reliable pool of resources that adapts to any workload or use case while offering a foundation for new data services.</li>
<li><a href="https://www.caringo.com/products/swarmfs.html" target="_blank">Caringo SwarmFS for Hadoop</a>—a native Hadoop connector that directly accesses the industry-leading massively parallel storage of Caringo Swarm, enabling Hadoop users to reduce their storage footprint while working with existing applications and meeting compliance regulations.</li>
<li><a href="https://www.caringo.com/products/filefly.html" target="_blank">Caringo FileFly™</a> for Swarm—plugs into file servers combining the performance of file servers with the scalability and economy of object storage without affecting existing mount points or applications.</li>
</ul>
In addition, we will be showcasing integration with Mangstor High-Performance Storage. Mangstor headquarters are a quick jaunt down the road from Caringo (an excuse to enjoy a drive down one of our most scenic roads in Austin, the Capital of Texas Highway). Their NVMe PCIe SSD and NVMe over fabric, all-flash storage solution accelerates application workloads over standard Ethernet or InfiniBand networks to provide 10X higher IOPs performance and 10X lower latency compared to legacy flash storage arrays. While Caringo software will run on any standard hardware, Caringo on Mangstor provides an integrated solution featuring low latency and high-throughput performance directly from the object store—so you can gain greater insights from your data, faster than ever before.
<a href="https://i2.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Mangstor.png?ssl=1"><img class="size-medium wp-image-2932 alignright" src="https://i2.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Mangstor.png?resize=300%2C240&ssl=1" alt="Supercomputing 2015 Caringo+Mangstor" width="300" height="240" srcset="https://i2.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Mangstor.png?resize=300%2C240&ssl=1 300w, https://i2.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Mangstor.png?w=500&ssl=1 500w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" /></a>Mangstor will also be at the SC15 expo, and we have joined forces to bring you an Apple Watch giveaway! Talk to an expert at either the Caringo (#489) or Mangstor (#3119) booth and receive an entry ticket. Complete the ticket, trek to the other side of the expo, and talk to an expert at the other booth. They will take your ticket and make sure it goes into the drawing.
<a href="https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Booth-1.png?ssl=1"><img class="alignleft size-medium wp-image-2929" src="https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Booth-1.png?resize=300%2C240&ssl=1" alt="Supercomputing Austin Caringo Booth" width="300" height="240" srcset="https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Booth-1.png?resize=300%2C240&ssl=1 300w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2015/11/SC15-Caringo-Booth-1.png?w=500&ssl=1 500w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" /></a>I’m leaving my home renovations in the hands of my hardwood flooring guy (who I happened to meet at my local pub) and I will be at SC15 (by the way, I’ve still got expo passes available if you need one; just email me at <a href="mailto:info@caringo.com">info@caringo.com</a>). Make sure you come by and have a beer with us Tuesday afternoon at the expo (our Founder and CEO Jonathan Ring and several of our VPs will be on hand to visit with you). We also invite you to connect with us on <a href="https://twitter.com/CaringoStorage" target="_blank">Twitter</a> and <a href="https://www.linkedin.com/company/caringo-inc-" target="_blank">LinkedIn</a>. And, I’ll be live tweeting from the event from my own handle, <a href="https://twitter.com/cooksarah9" target="_blank">@cooksarah9</a>.

The Caringo team is attending #SC15 this week. We will have demos for you so you can get up close and personal with our products, including Caringo Swarm, Caringo SwarmFS for Hadoop, and Caringo FileFly. Visit us at booth 489.

Sarah Cook is Caringo's Director of Marketing and brings deep expertise in Technical and Marketing Communications and Product Marketing. Sarah has worked for some of the biggest names in technology including Dell and Cisco. Sarah has a B.A. in English Composition with a minor in Music from the University of North Texas.

Sarah Cook

File Server (fileservers) provide a location for shared disk access so that data files can be shared between various end users and devices.

The Topic

Data Durability

Related Topics

Data Durability Recent Content

Metadata Storage Solutions

Crash Course on Object Storage at Virt-G Boston

Data Resilience & Recovery in Swarm Object Storage

Elastic Content Protection Technical Overview

Protecting Data with Caringo Swarm Object Storage

Object Storage Software, Beer, and Home Improvement