As if the data landscape was not already challenging, it became even more so because of COVID-19. With myriad companies suddenly moving to an almost completely remote workforce, the accessibility of files for remote workers was and remains on the critical path for just about every type of business you can imagine.
<h2>Challenges for Storing Data in the 21st Century</h2>
As Swarm release version 12 was in planning, we identified a number of challenges that organizations faced in dealing with data in this third decade of the 21st Century. As data piled up, traditional storage was often augmented with public cloud storage. For some, a public cloud solution was ideal, but for others, it was lacking in security, accessibility and affordability. So, with Swarm 12, we set out to make it easier for businesses to:
<ul>
<li>Add storage capacity continuously while transparently providing data access</li>
<li>Comply with industry and government regulations</li>
<li>Keep both TCA (total cost of acquisition) and TCO (total cost of ownership) within budget</li>
<li>Archive data securely long term</li>
<li>Ensure that remote staff can access the data they need</li>
<li>Leverage cost benefits of public cloud for disaster recovery</li>
</ul>
<h2>What’s New in Swarm Version 12?</h2>
In Swarm 12, we continue to evolve object storage into an intelligent data management system. From the beginning, we’ve worked to make Swarm smart (if you are not familiar with <a href="https://www.caringo.com/whitepapers/smarts-of-swarm">The Smarts of the Swarm whitepaper</a>, it is worth a read). And, in our latest release, we’ve focused on enabling more flexible distributed protection and immediate global content use across geographically dispersed sites. We’ve done this through a variety of changes including:
<ul>
<li>Automated management of distributed synchronous workflows via remote synchronous write (RSW) policies</li>
<li>Ease of use for end-users as they store, organize, find and share files through an updated user interface (UI) that includes drag & drop, multi-upload and an improved content view that is familiar to users</li>
<li>Expanded identity management connectivity via single sign-on (SSO) with SAML 2.0 that includes support for services like Okta, OneLogin and Google GSuite</li>
<li>Cost optimization for long-term (cold) storage leveraging tape through AWS S3 Glacier and AWS S3 Glacier Deep Archive support and certification with FUJIFILM Object Archive (a tape object solution)</li>
<li>Architectural optimizations for flash and optimal utilization of dense storage nodes</li>
</ul>
<a href="https://www.caringo.com/webinars/intelligent-data-management-access-archive-content"><img src="https://i2.wp.com/www.caringo.com/wp-content/uploads/2020/10/intelligent-data-management-access-archive-content.png?resize=640%2C360&ssl=1" alt="intelligent-data-management-access-archive-content" width="640" height="360" class="aligncenter size-full wp-image-20592" data-recalc-dims="1"></a>
<h2>Learn More About Swarm Version 12</h2>
Join me and Sr. Consultant John Bell for our Tech Tuesday webinar to learn more about <a href="https://www.caringo.com/webinars/intelligent-data-management-access-archive-content">Swarm 12 Intelligent Data Management for Access, Archive and Content</a>. We will take a deep dive into new features and functionality as well as the benefits that Swarm 12 brings to organizations struggling with storing and managing data. Feel free to bring your questions to the webinar, or email them to us at <a href="mailto:info@caringo.com">info@caringo.com</a> if you need more information immediately.
<a href="https://www.caringo.com/webinars/intelligent-data-management-access-archive-content" class="btn" title="Register to attend the live Swarm version 12 announcement webinar">Register now</a>

Swarm v12 is still object storage, but has evolved into an intelligent data management platform for content access, delivery and archive

Eric Dey, Caringo Director of Product, is our expert on Cloud Storage Infrastructure, and was one of the early developers on the Swarm Object Storage product. With a B.S. in Computer Science from the University of California Davis, Eric has extensive experience with cloud services, Linux open-systems, and applications. When Eric is not working on cloud storage solutions, he is often flying an airplane through the clouds as an active member of Austin Mooney Flyers, LLC.

Eric Dey

Object storage (object-based storage) is computer data storage architecture that manages data as objects, as opposed to file storage & block storage.

Object Storage

Swarm Storage provides a complete S3-compliant platform for data protection, management, organization & search at massive scale for unstructured data.

Swarm Storage

Caringo® SwarmFS lightweight file protocol converter brings S3-compliant object storage benefits to NFS &amp; SMB, seamlessly integrating file &amp; object.

SwarmFS

data management

Data Management Tool

Swarm 12 Intelligent Data Management for Content Access, Delivery & Archive

Caringo® SwarmFS lightweight file protocol converter brings S3-compliant object storage benefits to NFS & SMB, seamlessly integrating file & object.

<img src="https://i1.wp.com/www.caringo.com/wp-content/uploads/2017/06/New-In-Swarm.jpg?resize=300%2C169&ssl=1" alt="What’s New in Caringo Swarm?" width="300" height="169" class="alignright size-medium wp-image-9033" srcset="https://i1.wp.com/www.caringo.com/wp-content/uploads/2017/06/New-In-Swarm.jpg?resize=300%2C169&ssl=1 300w, https://i1.wp.com/www.caringo.com/wp-content/uploads/2017/06/New-In-Swarm.jpg?w=640&ssl=1 640w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" />Recently, we released updates to Swarm 9 with enhancements to improve the operation and ease of maintenance of your storage cluster. All customers with current support contracts can download the latest software from <a href="https://connect.caringo.com/" target="_blank" rel="noopener noreferrer">Caringo Connect</a>.
As the pioneer in object-based software-defined storage, we continue to evolve our technology to meet the emerging needs of our customers who span a diverse set of verticals from media & entertainment (M&E), to high-performance computing (HPC), to government, to managed service providers (MSP), just to name a few. Here are the highlights from our recent updates.
Metadata Typing Expanded
Those of you familiar with Caringo Swarm Object Storage understand the importance of our rich <a href="https://www.caringo.com/blog/medadata-for-business/" target="_blank" rel="noopener noreferrer">metadata</a> capabilities and how these capabilities enable organizations to manage and use data at a massive scale. We have expanded the built-in searching capabilities with a multi-representational metadata typing. Now, in addition to indexing content metadata as strings, Swarm now supports alternative representations—such as numeric, time, and geo-location—for the same metadata. This provides applications with powerful query capabilities when performing searching operations.
S3 Multipart Listing Compatibility Improvements
Increasingly, <a href="https://www.caringo.com/blog/nfs-s3-interoperable/" target="_blank" rel="noopener noreferrer">S3 interoperability</a> is a requirement for organizations. Contrary to AWS S3 API guidance, some S3 applications naively codify formatting validations for opaque response information in their multipart operations. In order to support these applications, the storage cluster now mimics the AWS formatting behavior of the S3 opaque response fields on multipart upload and listing operations. In addition, the Content Gateway includes additional improvements to support Amazon S3 compatibility.
Volume Upgrades and Management Simplification
Swarm volumes that hold mostly small objects and were formatted by Swarm 7.5 or earlier can now be automatically upgraded in place to better handle objects <100KB. This format upgrade allows the disk journal space to be correctly sized after-the-fact for existing deployments where small objects are the predominant storage and where the current journals are full. Additional management simplifications include improve handling for virtual machine volumes, encrypted volumes, and event reporting.
Want to learn more about what these enhancements mean and engage in live Q&A? Join me on June 21 for a live webcast. <a href="https://www.brighttalk.com/webcast/13173/264117" target="_blank" rel="noopener noreferrer">Register now</a> to watch live or view on-demand.
<a href="https://www.brighttalk.com/webcast/13173/264117" target="_blank" rel="noopener noreferrer">What’s New in Caringo Swarm Object Storage Webcast</a>

Recently, we released Swarm 9.2 with enhancements to improve the operation and ease of maintenance of your storage cluster.

Caringo uses Amazon S3 protocol for seamless policy-based tiering of data from primary storage to private Swarm cloud to public cloud.

Amazon S3

Founded in 2005, Caringo pioneered Object-Based Storage technology--forever changing the economics of storage with CAStor (now known as Swarm).

Caringo History

Caringo Swarm Object Storage protects digital assets, keeping them online & accessible for production workflows, active archive & on-demand streaming.

Media &amp; Entertainment

Metadata is "data about data" and can be descriptive, structural, administrative, reference or statistical metadata that is auto- or user-generated.

Metadata

What’s New in Caringo Swarm?

Media & Entertainment

<img src="https://i1.wp.com/www.caringo.com/wp-content/uploads/2017/10/Hybrid-Cloud.jpg?resize=276%2C201&ssl=1" alt="Hybrid Cloud Object Storage" class="wp-image-10057 alignright" width="276" height="201" srcset="https://i1.wp.com/www.caringo.com/wp-content/uploads/2017/10/Hybrid-Cloud.jpg?zoom=2&resize=276%2C201&ssl=1 552w, https://i1.wp.com/www.caringo.com/wp-content/uploads/2017/10/Hybrid-Cloud.jpg?zoom=3&resize=276%2C201&ssl=1 828w" sizes="(max-width: 276px) 100vw, 276px" data-recalc-dims="1" />To meet the business demands for infinite-scale storage and flexible access to data for their applications, IT organizations are utilizing object storage deployed on-premises, in public clouds, or a hybrid combination of both. While there are organizations that use public cloud for 100% of their storage and application infrastructure and there are those that will keep storage and applications completely on-premises for the foreseeable future, many are somewhere in the middle. They are exploring ways to incorporate public cloud storage as a component of their overall data storage infrastructure.
The first-order benefits of using a public cloud service like Microsoft Azure include off-site data protection as part of a disaster recovery plan or as a cost-effective way to store infrequently accessed archival data. Higher-order benefits of placing data into a public cloud come about when the cloud provider’s computational and analytics services are utilized as part of a comprehensive data processing and storage architecture.
So, what does that have to do with your storage provider? In short, you will be encumbered or empowered by their vision for data management and by how they architect their product to use public cloud.
When evaluating hybrid cloud off-site data protection, an important consideration is the tools the vendor has provided to specify which data should be sent to the cloud and which data should not. Due to internal policies or legal requirements, some data may not be appropriate to be sent to a 3rd-party provider. A hybrid-cloud storage system should provide the ability to designate when data is transmitted outside of the organization.
An equally important consideration is that, due to WAN bandwidth limitations, it may be infeasible to transmit the entire storage system to a public cloud. For example, sending 100TB of data over a WAN could easily take six months with a 100Mbps connection. And, for those organizations using a 1Gbps or faster connection, unless special routing equipment that contains TCP optimizations is used, the transmission rates may be no better.
Since bandwidth cost and availability are often barriers to transmitting huge data sets in a reasonable amount of time, public cloud providers offer physical shipment options to bulk-load the initial, and presumably largest, set of data into their clouds. If the data set is reasonably static and the future ingest rate is within an organization’s WAN bandwidth capabilities, these physical shipment options make sense. If, however, the data changes rapidly such that the WAN transmission cannot keep up, it will be infeasible to use a public cloud without the ability to be selective as to what data is transmitted and what data remains solely on-premises. In addition to transmitting data from an on-premises storage system to a public cloud, the storage system’s capabilities to recover your data in the event of corruption/loss in the on-premises system is also an important consideration.
A well-architected storage system allows an administrator to use the public cloud data set(s) as source to re-populate some or all the original on-premises data. This includes the possibility that the original on-premises hardware and its location are completely different. Additionally, the storage system should automatically determine which files are missing or corrupted and recover only those without needlessly transmitting and overwriting perfectly good files that are already in the on-premises system.
The architecture decisions and data management capabilities that a storage vendor incorporates into their product will determine if you are encumbered or empowered in managing your data. This affects your control over what goes into the cloud and your ability to use that data if it needs to come back to the on-premises storage.
In the second part of this article, I will cover the higher-order benefits of placing data into a public cloud and how to evaluate the hybrid-cloud functionality of an object storage system.

How are IT pros meeting business demands for infinite-scale storage and flexible data access? Object storage deployed in private, public and hybrid clouds.

Cloud architecture are the components needed for cloud computing. Typically, front-end and back-end platforms, cloud-based delivery and a network.

Cloud Architecture

Object-based storage (e.g., Amazon S3-compliant Caringo Swarm Object Storage) is the basis for all Cloud Storage⏤private, public or hybrid).

Cloud Storage

Hybrid Cloud is storage that has a mix of on-premises, private cloud storage (e.g., Caringo Swarm Object Storage) & 3rd-party public cloud services.

Hybrid Cloud

Caringo Swarm Object Storage can be used as part of a hybrid cloud solution Amazon S3-compliant cloud services such as Microsoft Azure.

Microsoft Azure

NFS is a distributed network file system protocol. Caringo SwarmNFS lightweight file protocol converter seamlessly integrates files & object storage.

NFSv4 is a distributed network file system protocol. Caringo SwarmNFS file protocol converter seamlessly integrates files & object storage.

NFSv4

Hybrid Cloud Object Storage | Part 1

<img src="https://i2.wp.com/www.caringo.com/wp-content/uploads/2017/10/Hybrid-Cloud-2.jpg?resize=376%2C262&ssl=1" alt="<a href="http://www.freepik.com/free-vector/business-background-design_1125316.htm">Designed by Freepik</a>" class="wp-image-10118 alignright" width="376" height="262" target="_blank" srcset="https://i2.wp.com/www.caringo.com/wp-content/uploads/2017/10/Hybrid-Cloud-2.jpg?zoom=2&resize=376%2C262&ssl=1 752w, https://i2.wp.com/www.caringo.com/wp-content/uploads/2017/10/Hybrid-Cloud-2.jpg?zoom=3&resize=376%2C262&ssl=1 1128w" sizes="(max-width: 376px) 100vw, 376px" data-recalc-dims="1" />Is your storage provider empowering you or encumbering you? This second part of the series discusses high-order benefits of placing data into a public cloud when your storage provider empowers you to use your data.
The <a href="https://www.caringo.com/blog/bloghybrid-cloud-object-storage-part-1/" target="_blank" rel="noopener noreferrer">first part of this article</a> showed the importance of providing tools for designating data that is sent to the public cloud with regard to off-site data protection. This ability to manage data sets by organizational units within the storage system and select data using object metadata combined with data management policies becomes crucial when going beyond hybrid-cloud basics.
When using public cloud for data tiering, the on-premises storage acts as a cache for data ingest and holds the working set data that is still active. Typically, data that is inactive or that meets other policy rules is moved into the public cloud for long-term storage using a least recently used replacement strategy. Even though some or all of the content may reside off-site, all of the data appears to reside locally and users retain full use of their data, including searching based on content metadata. (For more on metadata, <a href="https://www.caringo.com/blog/power-of-metadata/" target="_blank" rel="noopener noreferrer">read this article</a> by our CEO and Co-Founder Jonathan Ring.)
By using public clouds for tiering content, storage administrators can reduce the overall footprint of their physical hardware and take advantage of the public cloud provider for their scale-out capabilities. Simultaneously, the additional benefits discussed in this article are also realizable.
If backup and tiering will forever be your only goals for using public cloud, then the in-cloud data representation is not a primary concern for you or your users. However, the real power with public cloud platforms is in the data processing services that are available to multiply the value of your data. These high-order gains from public cloud are only possible when the representation of the data—including its metadata—are native to the cloud provider’s object storage system.
This native format for the in-cloud data means that your storage vendor must not use their own, homegrown data structures for the data and must not break it apart into pieces that are meaningless to the other cloud services. Your storage vendor should also preserve the organizational structure of the on-premises data when it is placed into the public cloud. This means that the multi-tenancy groups, containers/buckets, and even object names should be translated to the native format of the cloud provider’s system. Compromising on these will limit your ability to utilize the full power of the public cloud provider.
As mentioned, the real gains from using public clouds for more than bulk data storage come about as the other cloud services are brought to bear for data processing. The ability to designate data sets becomes crucial so that end-users and content owners, not just storage administrators, are able to direct the movement of data into the cloud so that additional tools can operate on it.
Example use cases for copying targeted data sets into a public cloud include:
<ul>
<li>CDN for high-bandwidth content delivery to customers</li>
<li>Image analysis and facial recognition</li>
<li>Video transcoding and scene analysis</li>
<li>Voice analysis and transcription</li>
</ul>
To design, host, and manage these use cases on-premises take more money and specialized personnel experience than many organizations can reasonably allocate. Selective data set movement to a public cloud combined with using these additional services allows for rapid application augmentation. Without this capability, many organizations would be limited in the variety of data analytics that they can use.
After data sets are processed in a public cloud, your storage vendor should provide the tools for your end-users and data owners to combine the results of the in-cloud data processing back into the storage system. Additionally, your users should be able to direct how the in-cloud data sets are handled after processing.
Does your storage provider encumber or empower you? Their vision to fit public cloud storage into their own storage product determines the answer to that question. Will you be able to do more than bulk storage with public cloud and use the data processing tools within the cloud? Look for storage systems that will address the first-order needs like offsite data protection and bulk storage and, simultaneously, empower your users to use the high-order data processing tools available within the cloud.
To learn more, <a href="https://www.caringo.com/webinars/hybrid-cloud-microsoft-azure/" target="_blank" rel="noopener noreferrer">watch our webinar</a> on demand.
<a href="http://www.freepik.com/free-vector/business-background-design_1125316.htm" target="_blank" rel="noopener noreferrer">Graphic contained herein is designed by Freepik</a>

Part 2 of our series on hybrid cloud object storage details the benefits of placing data into a public cloud when you are empowered to use that data.

Hybrid Cloud Object Storage | Part 2

<img src="https://i0.wp.com/www.caringo.com/wp-content/uploads/2018/01/Swarm9.4.jpg?w=300&ssl=1" alt="What's New in CaringoSwarm9.4" class="alignright size-full wp-image-10690" srcset="https://i0.wp.com/www.caringo.com/wp-content/uploads/2018/01/Swarm9.4.jpg?w=1400&ssl=1 1400w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2018/01/Swarm9.4.jpg?resize=300%2C171&ssl=1 300w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2018/01/Swarm9.4.jpg?resize=768%2C439&ssl=1 768w, https://i0.wp.com/www.caringo.com/wp-content/uploads/2018/01/Swarm9.4.jpg?resize=1024%2C585&ssl=1 1024w" sizes="(max-width: 1000px) 100vw, 1000px" data-recalc-dims="1" />The Caringo engineering team stayed busy through the holiday season to bring you the Swarm v9.4 update which includes new capabilities and improvements for storage administrators and end-users. Here’s a quick rundown of what is new in Swarm v9.4:
Access Policy Editor — The Content UI now includes an interactive policy editor for creating and updating access control policies that greatly simplifies content management for admins and end-users. The editor includes templates for the most commonly used policies while also providing fine-grained control over every aspect of policy management. Safeguard alerts are included that provide advisories if users may be unintentionally locking themselves out.
Dynamic Subcluster Reassignment — Swarm Storage now supports dynamic reassignment of subclusters which are used by admins for fault partitioning. With this change, nodes do not have to be restarted and the cluster remains continuously available to applications.
Elasticsearch Config Automation — Implementing Elasticsearch for Swarm requires making a set of configuration and settings file changes across all servers in the Elasticsearch cluster. Swarm now provides a configuration script that automates the process of consistently applying changes across all of the Elasticsearch servers.
OSS Updates — The embedded operating system in the Swarm Storage nodes is now based on the Debian 8 (codename “jessie”) distribution and the Linux kernel has been updated to version 4.4.84. These changes bring numerous operating system improvements and support for additional hardware such as the latest generation of 10Gbps network cards.
S3 Compatibility — Gateway includes several S3 protocol enhancements to track with Amazon S3 changes. Given that S3 is always evolving, Caringo monitors protocol changes and application trends to ensure maximum compatibility with 3rd-party applications and Swarm.
All customers with current support contracts can immediately download the latest <a href="https://connect.caringo.com/" target="_blank" rel="noopener noreferrer">software and documentation from Caringo Connect</a>. If you want to learn more about Caringo solutions or have any questions, <a href="https://connect.caringo.com/webform/contact-us" target="_blank" rel="noopener noreferrer">contact us</a>.

Learn about new capabilities and improvements for storage administrators and end-users in Caringo Swarm v9.4 Scale-Out Hybrid Storage.

Elasticsearch is a distributed, multitenant-capable full-text search engine with HTTP web interface; it is fully integrated into Swarm Object Storage.

The Author

Eric Dey

Author Related Topics

Stories by Eric

Swarm 12 Intelligent Data Management for Content Access, Delivery & Archive

What’s New in Caringo Swarm?

Hybrid Cloud Object Storage | Part 1

Hybrid Cloud Object Storage | Part 2

What’s New in Swarm v9.4