In part 2 of this 4-part series on evaluating enterprise data storage solutions, we provide an in-depth look at identifying the specific types of data your enterprise stores and choosing a data storage platform best suited for managing that data.
Block storage, object storage, and file storage: are the three primary architectures used to build custom data storage solutions that determine how data is processed, stored, organized, and retrieved. Each storage type has unique capabilities and limitations, which means enterprise data storage systems are not “one size fits all” solutions.
Short on time? Download our free Enterprise Data Storage Playbook — the Ultimate Guide to Finding a Solution to Help You Manage the Explosion of Unstructured Data
Comparing Data Storage Types
In the modern cloud age, object storage tends to be top of mind for many businesses, yet most data is created and consumed as files. Before you can effectively compare enterprise level storage solutions, it’s important to know the different types of data your enterprise stores, as well as how block storage, object storage, and file storage solutions each differ in their approach to data management.
What is Block Storage?
Block storage, also known as block level storage or elastic block storage, is a sequence of data bytes that contain a number of whole records that have a maximum length (a block size). The process of storing data into blocks is called blocking, and the process of retrieving data from blocks is called deblocking. Blocked data is generally stored in a data buffer, and read or written one block at a time, which is aimed at reducing overhead and speeding up the handling of the data-stream.
The truth is all storage is built on blocks. Enterprises must have chunks of data that are organized in a certain way, so that the chunks make sense if you try and read it from a protocol; everything is stored in the form of blocks. What’s most important is how that data is organized at the block level and how it’s accessed, which determines its type of storage.
Pros and Cons of Block Storage
One of the most notable pros of block storage is the ability to efficiently access and retrieve structured data from a database, usually over a storage area network (SAN) connection that uses different types of protocols like iSCSI and Fibre Channel, Direct Attached Storage (DAS), among others. Block storage is great for structured data due to the way legacy storage solutions do their metadata recall of where a certain block lives on the hard drive, using what’s known as journaling. The journaling system keeps track of all the data ever written over time. So, from a structured database perspective, if you’re reading and accessing data that way, it’s very fast.
Because block storage uses this chronological journaling system that adds more and more data over time, and each individual block of data can live independently across multiple environments, data requests are served quickly by retrieving and reassembling blocks from the most efficient path possible. While block storage presents an efficient and reliable method for managing structured data, it is unfortunately far less usable for managing unstructured data. Considering block storage is so limited in its ability to handle metadata, applications built around unstructured data will inherently struggle with metadata reliant operations including basic search and retrieval functions.
What is Object Storage?
Object storage, also called object-based storage, is an architecture that manages data as objects, a key difference when compared with a storage architecture like a file system. Enterprises can implement object storage at multiple levels, including: device level, system level, and the interface level. An object storage device enables the creation and management of shared and secure storage for enterprise storage networks.
Object storage was created to address storage architecture challenges, enabling capabilities like interfaces that are directly programmable by the application. This self-managed, shared and secure storage moves lower-level functionalities, such as space management, into the storage device itself, with access to the storage device by way of a standard object interface. Object-based storage also seeks to enable capabilities like a namespace that can span multiple instances of physical hardware, and data management functions like data replication and data distribution at object-level detail.
Pros and Cons of Object Storage
Object storage can work well for unstructured data in which data is written once and read once (or many times). Static online content, data backups, image archives, videos, pictures, and music files can be stored as objects. Databases in an enterprise object storage environment generally have data sets that are unstructured, which suggests the data will not require a large number of writes or incremental updates.
One of the challenges with object storage is that it is not ideal for transactional data, and furthermore, object storage was not created to replace NAS file access and sharing. Perhaps the biggest problem with object storage is that it does not support the locking and sharing mechanisms needed to maintain a single, accurately updated version of a file.
Our digital footprints create exponential increases in data and that data never gets thrown away. So understanding what data you have, how it’s growing over time, and what might have value (and what might not) becomes a difficult problem to solve, particularly with legacy scale-out and scale-up storage solutions.
To keep up with exponential data growth, legacy object storage solutions require enterprises to buy a big storage container, like an on-premises data center, but that doesn’t reveal anything about the data itself. As a result, some enterprises resort to buying off-the-shelf software that will help understand data, yet those tend to be very expensive and they don’t scale effectively—in fact, they fall apart when they reach a billion files or a petabyte of storage.
What can be deemed as a band aid solution for data management was for enterprises to start writing their own code to catalog all of the data they have, which is time-consuming and resource intensive. This problem was further compounded by the fact that while a data storage system itself knows immediately what’s inside it—because it’s part of its own data structure—any external software used could not provide real-time insights. Object storage systems simply do not scale well and cannot turn data into valuable information.
Fortunately, there is a scalable solution to enterprise data storage.
Enter file storage, where applying smarts to unstructured data is not just the future but a modern day reality; where scalability across on-premises and cloud environments meets cost-control thanks to fundamentally smarter unstructured data management.
What is File Storage?
File storage, also referred to as file-based storage (FBS) or file system, is a format or platform used to store and manage data as a hierarchical tree structured (as a file hierarchy), where files are identifiable in a directory structure.
File systems store data as a set of individual file paths, which are strings of characters used to uniquely identify the file in a directory structure. These unique identifiers include the file name, extension, and its path and are how a file system controls the storage, retrieval, and graphical display of the data for a user.
In layman terms, location is the umbrella term for path—as in “Search for the location of your data” or “Go to the location of your data”—which specifies how to find the file on the disk. Each file path also contains specific information, such as the file name, date of access, file directory, and more. Extensions indicate what kind of data is contained in the file, for example, .txt, .png, .java, .html, .doc, and so forth. A directory structure is the way in which a file system arranges files to make them accessible to the user.
Pros and Cons of File Storage
File storage systems are based on a block device as a level of abstraction for the hardware responsible for storing and retrieving desired blocks of data; however, the block size in a file system can be a multiple of the physical block size. This leads to lack of scalability and space inefficiency due to internal fragmentation, as file lengths are often not integer multiples of block size; thus, the last block of a file can remain partially empty. This creates fragmentation in which storage space is used inefficiently, thereby reducing capacity and performance.
Modern file storage systems like Qumulo pursued a resolution to this problem through a technique called Scalable Block Store (SBS), which from a block storage perspective, is the block layer of the Qumulo file storage system and its underlying mechanism to store data. The result is a file storage system with massive scalability, optimized performance, and data protection. In this way, unstructured data files can be extracted into a hierarchical file system type layout—combining the best of both file system architecture and block store architecture.
A key advantage of Qumulo’s file storage system architecture is the ability to predictively warm the cache to accelerate read performance with predictive prefetch—and drive down read latency. Qumulo’s file storage system compensates for some of the performance issues with block storage by using caching and other methods to make it faster. Qumulo Data Scientist, Tommy Unger, offers a brief demonstration in the video below, where he showcases its effectiveness with some real workloads on a cluster.
One of the main pros of the file system is the improved flexibility, supporting different types of workloads and the ability to scale. This is one of the main reasons why a file system is so great, because enterprises can scale-out and also scale-up; you can scale the storage far beyond the limitations of block storage. You can also scale to compute.
Legacy object storage solutions, on the other hand, will have cluster limitations. What they end up doing is having to essentially duct tape clusters together, because they’re using a block type of technology. Contrarily, Qumulo uses a file type of technology where data can be stored anywhere—in the cloud or on-premises, or both—and it’s more haphazard.
Qumulo’s file data platform, Qumulo Core, makes it simple to consolidate unstructured data with a single solution that provides real-time data visibility, automation, and ease-of-use to meet your performance, processing, and data retention requirements.
All this leads us to our next installment in this series, where we’ll offer a deeper dive into the primary differences between legacy file storage systems vs. modern file storage systems.
Stay tuned!
This article is only the second in a 4-part series in which we cover everything you need to know when evaluating enterprise data storage solutions—and has only scratched the surface on these important considerations. To learn more, download our new Enterprise Playbook for our most comprehensive guide on choosing the right data storage solution to help manage the explosion of unstructured data.
