<img class="alignleft size-medium wp-image-4214" src="https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/02/Swarm-8.png?resize=300%2C205&ssl=1" alt="Swarm 8 is Here" width="300" height="205" srcset="https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/02/Swarm-8.png?resize=300%2C205&ssl=1 300w, https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/02/Swarm-8.png?w=546&ssl=1 546w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" />When we founded Caringo over a decade ago, the largest <a href="https://en.wikipedia.org/wiki/History_of_hard_disk_drives" target="_blank" rel="noopener noreferrer">hard drive was 750 GB</a>, YouTube had just launched, the term ‘Cloud’ was used to describe weather and just storing and protecting millions of files was considered “next gen.” Fast forward 10 years to today, and 10 TB drives are here, a staggering <a href="https://en.wikipedia.org/wiki/History_of_hard_disk_drives" target="_blank" rel="noopener noreferrer">4 billion YouTube videos</a> are viewed every day, your grandma knows how to store data in “the cloud,” and even small organizations need to store, protect, organize and search billions of files. I stress the term need to because solving these issues is still a challenge for most, until today.
Today, we’ve consolidated a decade of experience and expertise to launch <a href="https://www.caringo.com/swarm-overview/" target="_blank" rel="noopener noreferrer">Swarm 8</a>—searchable storage for the cloud age—combining the ease of management and economics of cloud infrastructure with organization and search at scale. You no longer need to migrate data into disparate solutions for long-term preservation, delivery and analysis. Consolidate all files on Swarm, find the data you are looking for quickly, and reduce total cost of ownership by continuously evolving hardware and optimizing use of your resources.
So why is this important? Because forward-thinking businesses understand that the relationships between data and keeping content accessible and instantly searchable increase profit and agility. Said another way, if you can keep your data online, organize it and search it, you can continue to extract value from it. In the information age, those that can leverage their <a href="https://en.wikipedia.org/wiki/Long_tail" target="_blank" rel="noopener noreferrer">long tail</a> data will not only succeed, they will reap benefits orders of magnitude greater than those constrained by limits in traditional technologies.
This isn’t a new idea for us. It was what drove our original vision of changing the economics of storage. Even a decade ago, we knew the cost curve of storage wasn’t in line with what was needed to extract the value of data stored. And this cost curve becomes crippling at tens of millions of files…let alone billions of files. So we decided to change the cost curve and bring what was once attainable by only a few web-scale organizations with hundreds of developers to every organization.
Whether you are an IT admin tasked with delivering services internally or a developer tasked with enabling web apps for millions of users, Swarm 8 will help you achieve and exceed your business objectives. Here is a brief list of what’s new. This in no way encompasses all of the features in Swarm 8. These are just the highlights:
<ul>
<li>Dynamic insight and organization at massive scale. Gain robust insight and view relationships between objects through ad hoc search and query of object attributes or custom metadata.</li>
<li>Rapid file use case development. Object versioning and object renaming dramatically reduce the development effort of integrating <a href="https://www.caringo.com/blog/object-storage-keeping-velocity-change/" target="_blank" rel="noopener noreferrer">object storage</a> with file system-based applications and interfaces.</li>
<li>All-inclusive, no-surprise pricing. Swarm 8 comes standard with ALL functionality enabled. This includes unique Swarm features like Darkive™ adaptive power conservation, WORM, Integrity Seals, Legal Hold, multi-tenancy (for public/hybrid/private) cloud deployment, and Search.</li>
<li>Flexible deployment to meet any operational business requirements. Swarm 8 is available as software you run in your own data center(s), as a service in any of the <a href="https://www.caringo.com/blog/swarm-brings-amazon-s3-compatibility-microsoft-azure/" target="_blank" rel="noopener noreferrer">Microsoft Azure</a> globally distributed data centers or as a fully managed service within <a href="https://www.caringo.com/blog/bt-compute-storage-is-powered-by-caringo-swarm/" target="_blank" rel="noopener noreferrer">BT</a> or <a href="https://www.caringo.com/blog/telefonica-calls-on-caringo-swarm-for-future-proof-cloud-storage/" target="_blank" rel="noopener noreferrer">Telefónica’s</a> globally distributed data centers.</li>
</ul>
If you want to learn more, or want to ask our product experts questions, register for one of our upcoming Swarm 8 webinars:
<ul>
<li><a href="https://attendee.gotowebinar.com/register/4951425587571898881" target="_blank" rel="noopener noreferrer">February 18, 9 AM PST, 11 AM CST, Noon ET</a></li>
<li><a href="https://attendee.gotowebinar.com/register/9005998959809844993" target="_blank" rel="noopener noreferrer">March 1, 9 AM PST, 11 AM CST, Noon ET</a> 
</li>
<li><a href="https://attendee.gotowebinar.com/register/2376909215638055937" target="_blank" rel="noopener noreferrer">March 3, 10 AM GMT</a></li>
</ul>

Consolidate your files on Swarm and get instant access to your data. Caringo Swarm 8 is searchable storage for the cloud age.

With a rare understanding of technology, customers and business, Jonathan Ring brings over 33 years of experience to Caringo. Prior to founding Caringo, Jonathan was an active angel investor and VC investing and advising in a broad range of companies. Jonathan has run multinational distributed development groups, small R&amp;D teams and huge divisions at industry giants.

Jonathan Ring

Caringo uses Amazon S3 protocol for seamless policy-based tiering of data from primary storage to private Swarm cloud to public cloud.

Amazon S3

Caringo Swarm Object Storage provides an active archive (limitless & seamless pool of data) with data protection, management and search capabilities.

Cloud Architecture

Object-based storage (e.g., Amazon S3-compliant Caringo Swarm Object Storage) is the basis for all Cloud Storage⏤private, public or hybrid).

Cloud Storage

Caringo Swarm Object Storage can be used as part of a hybrid cloud solution Amazon S3-compliant cloud services such as Microsoft Azure.

Microsoft Azure

Object storage (object-based storage) is computer data storage architecture that manages data as objects, as opposed to file storage & block storage.

Object Storage

Software-defined storage (SDS) is data storage software for policy-based provisioning and management of data, independent of underlying hardware. 

Software Defined Storage

Swarm Storage provides a complete S3-compliant platform for data protection, management, organization & search at massive scale for unstructured data.

Swarm Storage

Search, Organization and Data Management in the Cloud Age

This article was originally published on <a title="Go to original article" href="http://linux.sys-con.com/node/3930199" target="_blank" rel="noopener noreferrer">Linux CloudExpo Journal</a>.
<img class="size-medium wp-image-5233 alignright" src="/wp-content/uploads/2016/05/Thumbnail_Images_May_JRing.png" alt="Bid Date Cloud Servers Isle" width="300" />Bridging the divide between legacy storage and new data management platforms could constrain IT organizations and budgets and could prevent the utilization of cost-effective scalable storage infrastructures. But, businesses can avoid some of these constraints by evaluating their storage options objectively and asking themselves three important questions.
A decade ago, we were putting 250-gigabyte drives into servers. When people mentioned the cloud, they were talking about the weather, and a business was considered to be on the cutting edge if it needed to store a few million files.
Now, we have access to 10-terabyte drives, and grandparents are using the cloud to store pictures of their grandkids. It’s now common for businesses to need access to billions of files, so companies need to move to newer systems to keep track of everything. With so many options available today, what’s really the best solution for storage?
Common Data Storage Systems 
To truly understand legacy storage systems, you need to know how storage has evolved beyond the hard drive. Here is a quick rundown of the most common solutions that have emerged over the years:
<ul>
<li>Storage Area Network (SAN): A SAN is a dedicated network that connects storage devices with servers typically using a Fibre Channel, InfiniBand, or Ethernet. SANs are commonly used for database servers and other applications that require a low-latency block-level storage interface. Advanced setups allow for clustering and failover capabilities among the servers.The downsides of SANs are that they often require exotic network hardware, proprietary software tools, and specialized staff to deploy and manage them. For these reasons, membership in the storage area network is normally limited to a small number of servers.</li>
<li>Network Attached Storage (NAS): The storage devices in a NAS can be a purpose-built NAS appliance or a general-purpose server running Windows or Linux that delivers files to clients. While there have historically been many protocols that connect storage devices and clients, the market has settled upon a couple: Network File System (NFS) and Server Message Block (SMB). A NAS appliance is an all-in-one bundle of integrated hardware and software that is built for the sole purpose of delivering files to clients. Almost any general-purpose server can also deliver files and act like a NAS with the appropriate level of administrative configuration.Unfortunately, there are inherent disadvantages of NAS systems. With limited potential to scale, they can quickly become costly, complex, and labor-intensive to manage.</li>
<li>Software-Defined Storage (SDS): SDS is still an evolving concept that can include file-based, object-based, block, cloud and storage management solutions. Software-defined storage essentially separates the data and services layers from the underlying hardware. Software-defined storage solutions typically involve storage virtualization, and they may provide features like search, organization, replication, distribution, thin provisioning, snapshots and backup to name a few.</li>
<li>Cloud-Based Storage (Public and Private): As a multi-tenant environment, a public cloud storage system requires you to purchase a portion of a cloud-based computing environment that is shared with many other tenants. Public cloud storage is offered in an on-demand arrangement with monthly payments that can be advantageous; however, capacity and access costs are compounded monthly and won’t go down until data is deleted. Because you pay for the bandwidth to use your data, you may resist running analytics and other operations that would incur additional monthly charges. Private cloud storage solutions let you deploy storage as a service within your data center. You need to make an upfront capital investment in hardware and have the data center space and electrical power to run the service. If security is a priority, you are storing large amounts of data for long periods of time, or you are performing a lot of reads (such as analytics) on your data, a private cloud is almost always the best option. There are also hybrid solutions that provide a combination of private and public services.</li>
<li>Object-Based Storage: One popular type of SDS is object-based storage, which is at the heart of many public and private cloud-based storage services. In this model, there is no hierarchical folder structure; however, object-based storage does provide a method for data organization using metadata (often defined as “data about data”). In object-based storage systems, the data is organized into self-contained entities (objects). This flat approach provides for greater scalability and can be less expensive than block or file-based storage systems. For businesses with a need to store and search through high volumes of data, this is often the ideal solution.</li>
</ul>
Building a cost-effective and scalable storage infrastructure is not a task to be taken lightly. Initiatives like this have the potential to impact IT resources and inflate budgets. So how do you bridge the divide between legacy storage systems and the new data management platforms?
Planning for the Future 
Bridging the gap between legacy storage and newer technologies sometimes requires ensuring compatibility through protocols such as <a href="https://www.caringo.com/blog/blogblogfollow-the-leader-amazon-s3-compatibility/" target="_blank" rel="noopener noreferrer">S3</a>, RESTful HTTP, NFS, and SMB. As a result, business and IT leaders should consider a few important matters before determining the best data management platform to use for taking their businesses into the future.
<ol start="1">
<li>What type of data is being stored, and how quickly is it growing? SAN and NAS are still your best options for structured data; however, the total amount of structured information an organization has is often <a href="http://www.ibmbigdatahub.com/blog/do-not-ignore-structured-data-big-data-analytics" target="_blank" rel="noopener noreferrer">less than 10 percent</a>. Unstructured data is often 90 percent or more of the total capacity need. If you focus on your unstructured data growth rate year over year, you’ll most likely notice an acceleration.Some of this acceleration can be accounted for in factors such as the improvements in resolution for videos and images as well as new sources of unstructured data, such as log files, metrics, and data created by devices. Create a formula based on these considerations, and use it in conjunction with your historic storage capacity compounded annual growth rate (CAGR) to estimate your needs three to five years out. Using your forecasted capacity need, select a storage solution that can expand to accommodate your expected growth.</li>
<li>What are your access patterns? When you think about access, consider what (device, application, etc.) and who needs access and exactly how they will access it (e.g., geographical location and interface or search mechanism). When you have billions of files, how will you find what you need? Almost as important, how will you determine what you don’t need so you can confidently delete this data? When choosing your future storage platform, make sure your chosen solution supports your organization’s access requirements.</li>
<li>How long must the data be retained? Data retention rates vary by industry from a few seconds to indefinitely. When you think about retention, consider the cost of different protection methods versus the value of the data and ease of migration (e.g., how easy it is to continue to evolve the underlying hardware infrastructure). If you factor ease of migration into your decisions today, you will make your life simpler when you one day find yourself needing to migrate petabytes or possibly exabytes of data.Beyond how long you are required to retain data, consider how long that data may be valuable to you from both an information and a monetary perspective.</li>
<li>The relationships between data and keeping content accessible and instantly searchable increase profit and agility, something every forward-thinking business leader understands. If you can keep your data online, organize it, and search it, you can continue to extract value from it.</li>
</ol>
In the information age, those who can leverage long-tail data will not only succeed, but they will also reap benefits in orders of magnitude greater than those constrained by the limits of traditional technologies.

How will you bridge the divide between legacy storage and new data management platforms? Check out these tips on planning for the future.

Network attached storage (NAS) is a high-performance file-level data storage device connected to a computer network that provides data access.

Unstructured data refers to digital files with no pre-defined data model or organizational structure (e.g., video, medical imaging & photos).

Unstructured Data

The Future of Data Storage Solutions

<img class="alignright" src="/wp-content/uploads/2017/03/video-surveillance-security-today.jpg" alt="How to Store Video Surveillance More Effectively, Part 1" width="282">As the amount of video data grows and video resolution improves, the requirement for storage space increases and it becomes more difficult and expensive for law enforcement agencies to store videos, even for a short time.
However, securely storing large amounts of video data and being able to find the data easily is now a requirement for law enforcement agencies. Legal procedures and legislation like the <a href="https://www.foia.gov/" target="_blank" rel="noopener noreferrer">Freedom of Information Act</a> require the secure transfer of full or partial government-controlled data if requested.
Even if you do have the storage space, dumping a massive amount of files into a data silo and not attempting to <a href="http://www.govtech.com/opinion/Metadata-Your-Citys-Secret-Weapon.html" target="_blank" rel="noopener noreferrer">index or organize them</a> in any way can render important video footage unsearchable and, therefore, useless.
The Video Storage Problem
The key elements to consider in determining storage requirements for video surveillance are the <a href="http://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Video/IPVS/IPVS_DG/IPVS-DesignGuide/IPVSchap4.html" target="_blank" rel="noopener noreferrer">“three R’s”: retention, reliability, and resolution</a>.
<ul>
<li>Retention times vary by state and range from 30 days for non-evidence to seven years to indefinitely for certain crimes. The longer the retention time, the more storage a law enforcement agency needs.</li>
<li>Reliability is something most in the storage industry don’t think about. Storage and storage media will eventually fail. You need to protect the data in some way, and that comes with additional storage overhead ranging from 20 percent to 300 percent, depending on the level of availability needed.</li>
<li>Resolution is what most people consider when they think of video size, and that is driven by frames per second. While cameras in police cars record <a href="http://www.theiacp.org/portals/0/pdfs/InCarCamera.pdf" target="_blank" rel="noopener noreferrer">1.5 gigabytes per hour</a> at standard definition, body-worn cameras shooting at 720 HD consume about 2.5 gigabytes per hour. Increase this resolution to 1080 HD, and the storage consumption increases to 5 gigabytes per hour. Now, multiply body-worn camera storage needs by the <a href="http://www.nleomf.org/facts/enforcement/" target="_blank" rel="noopener noreferrer">over 900,000</a> sworn law enforcement officers in the U.S. alone, and you’re looking at a staggering amount of data.</li>
</ul>
Legacy storage systems are <a href="http://www.policeforum.org/assets/docs/Free_Online_Documents/Technology/implementing%20a%20body-worn%20camera%20program.pdf" target="_blank" rel="noopener noreferrer">incredibly expensive</a>, especially considering the retention time a piece of evidence may require. One police department spends $111,000 annually to store evidence associated with just 50 cameras. Other departments are purchasing even more storage after realizing that they’re already <a href="http://www.computerworld.com/article/2979627/cloud-storage/as-police-move-to-adopt-body-cams-storage-costs-set-to-skyrocket.html" target="_blank" rel="noopener noreferrer">one-third</a> of the way through the storage they pre-purchased.
And while cloud storage may work for smaller departments, the compounding costs over time and the bandwidth necessary to move data to the cloud need to be considered. As a result, some departments are looking for public and private cloud storage that is economical; the answer is object storage.
[Editor’s note: Want to learn more? Watch for part 2 of this article next week, and join Caringo Product Manager Ryan Meek for <a href="https://www.caringo.com/webinars/object-storage-digital-evidence-security/" target="_blank" rel="noopener noreferrer">Object Storage for Digital Evidence & Security</a> on March 22. <a href="https://www.caringo.com/webinars/object-storage-digital-evidence-security/" target="_blank" rel="noopener noreferrer">Register</a> now to watch live or on-demand after the event.]
<a href="https://www.caringo.com/webinars/object-storage-digital-evidence-security/" target="_blank" rel="noopener noreferrer"><img class="alignnone wp-image-7509 size-medium" src="https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/10/Watch-Recording-01.png?resize=300%2C58&ssl=1" alt="Watch the Recording" height="58" width="300" srcset="https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/10/Watch-Recording-01.png?resize=300%2C58&ssl=1 300w, https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/10/Watch-Recording-01.png?resize=768%2C148&ssl=1 768w, https://i2.wp.com/www.caringo.com/wp-content/uploads/2016/10/Watch-Recording-01.png?w=950&ssl=1 950w" sizes="(max-width: 300px) 100vw, 300px" data-recalc-dims="1" /></a>
This article was originally published by <a title="Security Today Website" href="https://securitytoday.com/" target="_blank" rel="noopener noreferrer">Security Today</a>.

The requirement for storage space is ever increasing for video surveillance. Learn how to handle that volume more effectively. Part 1 of 2.

Swarm Object Storage continuously monitors data to ensure multiple copies are available at any given time, so you always have a back-up copy.

Backup &amp; Recovery

Law enforcement (those who enforce laws, investigate crimes and make arrests), can use Object Storage to store and protect video surveillance data.

Law Enforcement

Video Surveillance

How to Store Video Surveillance More Effectively, Part 1

Backup & Recovery

<img class="alignright" src="/wp-content/uploads/2017/03/evidence-and-security-webcast.png" alt="How to Store Video Surveillance More Effectively, Part 2" width="282" />The Benefits of Switching to Object Storage Systems
Object storage systems address specific concerns in law enforcement associated with data privacy, secure access, reliability, and economical scale. Some of the biggest worries in storing data are the possibilities that data could be lost, corrupted, stolen, or easily accessed (making privacy a concern), that existing storage could be depleted, and that content could be difficult and time-consuming to locate.
These systems store video keys with value pairs so there is a unique identifier for each video rather than a server name, series of directories, and file name, making it easier to locate specific files. All you need is the unique ID. Object storage systems are also economical because they run on standard servers, and in the event of a change or update in technology, data can be easily migrated to newer, more efficient hardware.
Characteristic of all object storage solutions is that they scale in a single location or multiple locations, they are highly automated, and they include “self-healing” technology. And best-of-breed solutions include compliance features like write once read many, legal hold, and integrity seals that can be used in courts of law to demonstrate a video hasn’t been tampered with.
Storing your video data with object storage makes it more useful because it:
<ol>
<li>Consolidates data in an easy-to-search, resilient storage solution. Storing data in disparate silos means that when you need to find a file, it’ll be like finding a needle in a haystack, which can lead to heavy penalties and fines. Files need to be consolidated and have metadata stored directly with the file so that they will be easily searchable.</li>
<li>Ensures your storage is admissible in a court of law. Agencies need to make sure their storage meets compliance standards with features like WORM, legal hold, and integrity seals to prove in a court of law that <a href="https://icoportal.s3.amazonaws.com/files/JQ7j3yKk/Whitepaper-Data-Protection-2.pdf" target="_blank" rel="noopener noreferrer">digital evidence hasn’t been tampered with</a>.</li>
</ol>
These features can prevent data from being modified or deleted, and they can also protect the original data set. Replicas of the data are made in the event of losses or malfunctions, and when transferring data, integrity seals can prevent tampering or transmission errors. Even in the event of catastrophic failures, data can be recovered from a remote location with ease. All these features mean that there will be no question that the videos presented in court are accurate.
<ol start="3">
<li>Makes sure your storage solution can scale out as your video needs grow. This accommodates growth in the number of cameras used, resolution of capture, retention rates, and overhead for proper resilience and protection. As your data grows, your storage will need to grow with it.</li>
</ol>
To scale out video-surveillance capabilities, law enforcement agencies have a number of choices, such as software-defined solutions and cloud technologies. Agencies need to choose whether to use a public or private cloud and then look at the capabilities they require. Prepare for the future today, and make sure your solution can scale to meet your needs for the next 5 to 10 years.
Law enforcement officials, and the lawyers and judges who work with them, need to find smarter ways to consolidate, protect, and search data being stored. To achieve this, data must remain online, accessible, and searchable with proper metadata attached. Object storage is one of the most cost-effective and secure ways to meet those requirements.
[Editor’s note: Want to learn more? Make sure to join Caringo Product Manager Ryan Meek for <a href="https://www.brighttalk.com/webcast/13173/248785" target="_blank" rel="noopener noreferrer">Object Storage for Digital Evidence & Security</a> on March 22. <a href="https://www.brighttalk.com/webcast/13173/248785" target="_blank" rel="noopener noreferrer">Register</a> now to watch live or on-demand after the event.]
This article was originally published by <a title="Security Today Website" href="https://securitytoday.com/Articles/2016/10/07/How-to-Store-Video-Surveillance-More-Effectively.aspx?Page=1" target="_blank" rel="noopener noreferrer">Security Today</a>.

Object storage systems address specific concerns in law enforcement associated with data privacy, secure access, reliability, and economical scale.

Big storage establishes a vast data pool (e.g., Caringo Swarm Object Storage) free of file hierarchies & restrictive load limits, accessible by HTTP.

Big Storage

With Caringo Swarm Object Storage, you get continuous, built-in data protection along with exceptional resilience, recovery, and content integrity. 

Data Protection

Scaling from small to 100s of PBs is one of the hallmarks of the Swarm pure object storage approach and a testament to our longstanding installations.

The Author

Jonathan Ring

Software Defined Storage & Author Related Topics

Software Defined Storage Stories by Jonathan

Search, Organization and Data Management in the Cloud Age

The Future of Data Storage Solutions

How to Store Video Surveillance More Effectively, Part 1

How to Store Video Surveillance More Effectively, Part 2