Enabling Object Storage Efficient File Movement

Pairing Partial File Restore with SMB, NFS, HTTP and Amazon S3 protocols enables digital video to reap the benefits of object storage

Increase Quality, Increase Efficiency, Increase Speed, Decreased Cost and TCO
Increase Quality, Increase Efficiency, Increase Speed, Decreased Cost and TCO

One of the most popular uses of Swarm is for storing, archiving and delivering

digital video.

In a recent blog, I detailed the market conditions that are driving customer requirements. The blog was about sports video; however, we are seeing the same trends in every market that relies on digital video (post production, film, surveillance, houses of worship and corporate training to name a few).

Increase Quality, Increase Efficiency, Increase Speed, Decreased Cost and TCOThe biggest challenge for many professionals today is enabling “on-demand” in existing workflows. Said another way, they are struggling with providing instant access to digital video and delivering it immediately to any device. At the heart of enabling “on-demand” is efficient file movement. This blog details a few of the specific features and interfaces Swarm employs to enable efficient file movement. The first step in efficient file movement is being able to integrate with Swarm via your existing applications and workflows.

Jump To:


Traditional Gateway Connector: NFS clients/servers, FUSE, namespace, spooling, caching, proxy, object store vs. SwarmNFS: nfs clients, swarm object storage

Sustained Data Streaming for NFS & SMB with SwarmNFS

Unlike many gateway and file system interfaces on the market, SwarmNFS is a file-to-object converter. It provides a mountable volume for your NFS or SMB applications and converts files to objects in a lightweight fashion (in flight) without spooling or caching. Therefore, you can use NFS, SMB or even S3 to read, write, modify or access a file, enabling true multi-protocol access. In a recent benchmark, a single instance of SwarmNFS delivered 1.56 GB/s read performance on commodity hardware.

Continuous S3 Support

The Amazon S3 API has become the de facto object storage interface. I stress “de facto” because it is technically not a standard. That said, we spend a lot of time making sure we stay as true to the specification as possible.

Why is S3 Support Important?

Storage Secrets: What are storage vendor isn't telling you about amazon aws S3S3 support is important because beyond standard storage protocols like NFS and SMB, the S3 interface is the way that most ISVs, data movers, and asset managers integrate with the cloud and on-premises object storage. Caringo’s object-based software-defined storage solution Swarm supports the Amazon S3 API through an extensible architecture, which later can be used to seamlessly support additional third-party APIs. A broad range of applications that currently support the Amazon S3 API work directly with Swarm. If you are interested in learning more about how Swarm plugs into asset management solutions via the S3 API, you should attend our webinar How to Enable Video On-Demand in Workflows on May 30 or watch afterwards on demand. Caringo will be demoing all of the asset manager integrations we highlighted at this year’s NAB 2019 Show, including CatDV, Marquis Project Parking, Cantemo and Vidispine. Or Caringo’s webinar on May 28 What Your Storage Vendor Isn’t Telling You About S3, where Caringo industry experts will discuss the secrets and important details your storage vendor isn’t telling you.

Parallel Uploads

From an architecture perspective, Swarm employs a parallel approach—that is, all nodes can perform all operations. This makes multi-part or “parallel” uploads an efficient way to ingest files and it also streamlines combining the multiple parts of a file once on Swarm.

Range Reads

The file movement benefits of Swarm aren’t just on ingest, but also provides efficient ways to access data, as the native interface to the software is based on HTTP. Swarm object-based storage device and software enables range reads, offering an application like a video player; the ability to specify the exact location of a file to start a playback operation. This eliminates the need to download or cache the undesired portion of the file.

Partial File Restore

Partial File Restore in Swarm (currently in beta with general release scheduled for Fall 2019) is a well-known feature in the M&E world that Caringo is bringing to object storage. Partial File Restore for object storage enables the ability (via a web-based UI or the API) to specify a portion of a video file you want and then create a clip of only that portion. That clip can then be moved to a specific application, downloaded by authorized users, or streamed directly from Swarm to authorized users, employees, subscribers or viewers. Get a personalized preview of Partial File Restore before Caringo’s launch later this year.

Wrap Things Up

This is just a short list of interfaces and features that enable digital video professionals to leverage the benefits of Swarm software-defined storage for efficient file movement into, within and out of Swarm while plugging into existing workflows. In addition to the resources I listed above, we have a growing library of on-demand webinars and highly informative blogs. With the Swarm 11 release just around the corner, our field-hardened object storage contains far more features than the few highlighted here. If you are interested in a full overview, don’t hesitate to reach out to Caringo with questions about your specific use case or to schedule a private demo!

Caringo Swarm Logo Caringo SwarmNFS Lightweight File Protocol Converter
Adrian Herrera
Adrian Herrera

About The Author

Adrian "AJ" Herrera has over 20 years of experience bringing innovative storage, cloud and media software and services to market. With an MBA from San Diego State University and a BS in Information Systems from Chapman University, AJ serves as Caringo's VP of Marketing.


More From Author