In this 4-part series on evaluating enterprise data storage solutions, we will provide you with an overview of the storage options available today, compare those solutions, and help you choose an ideal storage solution based on the types of data your enterprise stores. Additionally, this series will help determine if a scale-out network attached storage (NAS) solution is the best path for your business—with specific, real-world examples that enterprises requiring High-Performance Computing (HPC) encounter inside data lifecycles and how they transformed data from a raw state to a useful one.
How Efficient Is Your Enterprise Data Management?
Maintaining an enterprise IT architecture is like owning an old car that is constantly in the repair shop: the incremental costs add up, and the resources consumed could be invested in a newer model for a better return. Likewise, if you are an IT systems administrator constrained to storage technology built on monolithic, proprietary hardware that is inefficient, costly, and difficult to manage, it may leave you struggling to not only catch up but also to support data transformation initiatives.
When on the hunt for a scalable enterprise data storage solution, it’s crucial to understand whether the storage you choose is designed for working with data and applications in its native form. We’ll cover this in more detail below, as well as outline some of the main considerations when evaluating your HPC workflows. All told, this will help guide your decision in choosing a solution that best fits your enterprise’s needs today and in the future.
Evaluate Your High-Performance Computing Workflows
Most data originates in files, created and accessed directly from native applications or mounted file systems. Working with this file data natively means accessing it over industry standard protocols like Network File System (NFS), Server Message Block (SMB), or direct File System pass (block).
Data stored in its native format is considered unstructured data, meaning it lacks a pre-defined data model or schema and cannot be be stored in a traditional relational database (more on this later). Because this kind of unorganized data can’t simply be stored in a set of tables using columns and rows, enterprises have traditionally struggled with the fundamental challenge of managing, analyzing, and leveraging their unstructured data in a meaningful way, due to the complex and time-consuming data analytics processes required for extracting valuable insights.
Analysts at Gartner estimate that unstructured data represents an astounding 80 to 90% of all new enterprise data. This might sound surprising, but the reality is that enterprise data has been predominantly made up of unstructured data for decades now. In fact, in 1998 Merrill Lynch claimed, “Unstructured data comprises the vast majority of data found in an organization, some estimates run as high as 80%.” Lynch essentially made the assertion that as the volume of worldwide data creation continues to grow year over year, the more important it will be for highly scalable enterprise data management solutions that can effectively leverage this data in a meaningful way.
This “explosion of unstructured data” is being generated from video cameras, recording devices, satellites, sensors, genomic data, aerial imagery, and other IoT connected technologies—and represents a potential gold mine of insights.
Are You Leveraging Your Data in its Native Form?
Successful enterprises are storing, managing, and building High-Performance Computing (HPC) workflows and applications with file data in its native form—leveraging locally mounted file systems (made accessible by creating NFS exports and SMB shares) and data services that are natively integrated with cloud object stores (like Amazon S3 and Microsoft Azure)—and transforming that data into value. These innovators are embracing and managing data in all its forms to create new business models, medical treatments, consumer products, business intelligence tools, and digital media.
Can You Track and Manage Your Unstructured Data?
For many HPC corporations leveraging legacy storage and cloud-native applications, the task of processing, managing, and transforming unstructured data from file to object is a huge challenge. Most technology is not built to solve this problem, meaning companies must rebuild their architecture, refactor applications, or use third-party data movement packages to generate value from their data—in many cases this leads to vast data silos with little visibility into this data. Furthermore, organizations are frequently limited to only certain protocols that may not be supported or appropriate for certain applications or end users. The difficult outcomes for many leading corporations around the world are that this valuable data is never used, is inefficiently accessed, and is often poorly understood.
In a 2019 NewVantage Partners’ Big Data and AI Executive Survey, consisting of 64 C-level technology and business executives representing very large corporations, 53% of survey respondents say “they are not yet treating data as a business asset.” These alarming results come in spite of 92% of respondents reporting that the pace of their Big Data and Artificial Intelligence (AI) investments is accelerating.
Assess Your Specific Enterprise Data Storage Needs
Enterprises needing to enable large datasets in HPC environments with unstructured data means that having the ability to process and serve data is part of their business. To that end, when considering an optimum enterprise data storage solution, it’s important to evaluate whether it will meet your capacity, performance, data integrity, and scale-out requirements needed to process data and serve potentially dense and high-performance workflows.
Evaluate Enterprise Data Storage Solutions Ideal for Your HPC Workflows
An optimum enterprise data storage solution should provide the infrastructure needed to leverage HPC resources in their workflows. According to a Forbes survey, more than 95% of businesses face some kind of need to manage unstructured data, and more than 150 trillion gigabytes of data will need analysis by 2025—meaning file storage is becoming more important than ever.
Efficient Unstructured Data Management
Given that unstructured data represents most of all new data created every day, the more efficiently HPC companies can consolidate, process and leverage this data, the more successful their outcomes will likely be. It’s no surprise, then, that an ideal enterprise data storage solution is designed for working with this type of data natively.
Object Storage vs. File Storage
In the modern cloud age, object storage tends to be top of mind for many businesses, yet most data is created and consumed as files. Object storage is an architecture that manages data as objects, as opposed to a storage architecture like a file system. File storage is a format or program for storing and managing data as a file hierarchy, in which files are identifiable in a directory structure (generally displayed as a hierarchical tree structure).
File systems provide the fundamental abstraction of hierarchy that enables computers and humans to operate on semantically interesting groupings of data. Sure, enterprise data storage users appreciate having one big bucket of storage. However, object storage systems present a host of unforeseen, next-generation problems; for instance, object storage is not as performant.
Get the Guide: Download the Enterprise Data Storage Playbook
Assess Your Unstructured Data Management Needs
Processing petabyte-scale data requires the right enterprise data storage solution based on the type of data you need to analyze. For example, to process and analyze unstructured data that exists in the cloud and on-premises, companies would need a file data platform that can meet the demands of a hybrid storage infrastructure while also providing real-time analytics and insights. When evaluating enterprise data storage types, it’s more important than ever to choose the solution that best fits your enterprise’s needs today—and in the future.
Align Your HPC Workflows With a Modern Enterprise Storage Solution
Legacy File Storage Systems
Legacy file storage systems are based on a block device as a level of abstraction for the hardware responsible for storing and retrieving desired blocks of data; however, the block size in a file system can be a multiple of the physical block size. This leads to lack of scalability and space inefficiency due to internal fragmentation, as file lengths are often not integer multiples of block size; thus, the last block of a file can remain partially empty. This creates fragmentation in which storage space is used inefficiently, thereby reducing capacity and performance.
Legacy Object Storage Systems
Some companies are attempting to adopt legacy object storage systems as a solution to the scale and geo-distribution challenges of unstructured data. However, adopting object storage in use cases for which it was never intended is a poor technical fit. In order to achieve this, object stores intentionally trade-off features that many users need and expect: transactional consistency, modification of files, fine-grained access control, and use of standard protocols such as NFS and SMB, to name a few. Object storage also leaves intact the problem of organizing data; instead, encouraging users to index the data themselves in some sort of external database. This may suffice for the storage needs of stand-alone applications, but it complicates collaboration between applications, and between humans and those applications.
A surprising amount of valuable business logic is encoded in the directory structure of enterprise file systems. Therefore, the need for file storage at scale remains compelling.
Modern HPC workflows
Modern HPC workflows almost always involve applications that were developed independently yet work together by exchanging file-based data, an interop scenario that is simply not possible with object storage. Furthermore, object stores don’t offer the benefits of a file system for governance.
Modern File Storage Systems
Modern file storage systems such as Qumulo Core, sought out to solve this problem through a technique called Scalable Block Store (SBS). The Qumulo file system is built on the SBS, a virtualized block layer, which uses the principles of massively scalable distributed databases and is optimized for the specialized needs of file-based data.
From a block storage perspective, the SBS is the block layer of the Qumulo file system and its underlying mechanism to store data, giving the file system massive scalability, optimized performance, and data protection. Time-consuming work such as protection, rebuilds, and deciding which disks hold which data occurs in the SBS layer, beneath the file system. In this way, unstructured data files can be extracted into a hierarchical file system type layout—combining the best of both file system architecture and block store architecture.
The virtualized protected block functionality of SBS is a huge advantage for the Qumulo file system. Because the Qumulo file system uses block-based protection, small files are as efficient as large files. The result is a file system with unmatched scale characteristics. In contrast, legacy storage appliances were simply not designed to handle the massive scale of today’s data footprint, which use inefficient mirroring for small files and system metadata.
Is Scale-Out Network Attached Storage (NAS) the Future of Enterprise Data Storage Management (EDM)?
Legacy scale-up and scale-out file systems are not capable of meeting the emerging requirements of managing storage on-premises and in the cloud at scale. The engineers who designed them 20 years ago never anticipated the number of files and directories, and mixed file sizes, that characterize modern workloads. They could also not foresee cloud computing.
The Rise of Unstructured Data
Enterprises increasingly rely on unstructured data storage management (EDM) for regulatory, analytics, and decision making. Unstructured data is the backbone of analytics, machine learning, and business intelligence.
Enterprise Data Management (EDM) Requires Scalability
Enterprises needing to enable large datasets in HPC environments with unstructured data means that having the ability to process and serve data is part of their business. For this reason, enterprise IT systems and storage administrators are seeking a solution designed for working with this type of data natively. The ideal storage solution for this will meet their capacity, performance, data integrity, and scale-out requirements needed to process data and serve potentially dense and high-performance workflows.
Scalable Enterprise Data Storage Solutions with Scale-Out NAS
Qumulo was founded in 2012, as the crisis in file storage was beginning to reach its tipping point. A group of storage pioneers, the inventors of scale-out NAS, joined forces and formed a different kind of storage company, one that would address these new requirements head on. The result of their work, and the team they assembled, is Qumulo, which developed the world’s first enterprise-proven, hybrid cloud file storage system that spans the data center, the private clouds and the public clouds. It scales to billions of files, costs less, and has a lower Total Cost of Ownership (TCO) than legacy storage solutions. Real-time analytics let administrators easily access and manage data regardless of size or location. Qumulo’s continuous replication enables data to move where it’s needed, when it’s needed; for example, between on-premises and clusters running in the cloud or between clusters running on different cloud instances.
Choosing the Right Enterprise Data Storage Solution
With this brief overview of how to evaluate enterprise data storage solutions and compare those solutions, you should now have a better understanding of how to choose an ideal data storage solution based on the types of data your enterprise stores. For more insights, see part-2 in this series in which we provide a more thorough comparison of the different data storage types: block storage vs object storage vs file storage.
This article is only the first in a 4 part series on Why Enterprises Should Consider File Data When Evaluating Enterprise Data Storage Solutions—and has only scratched the surface on these important considerations. To learn more, download our new Enterprise Playbook for our most comprehensive guide on choosing right data storage solution to help manage the explosion of unstructured data.
STAY TUNED!
Stay tuned for parts 3 and 4 in this series where we will evaluate and compare legacy versus modern file storage systems, and then discuss how the Qumulo Scalable Block Store (SBS) has revolutionized the enterprise data storage industry with a state-of-the-art file storage system that provides massive scalability, optimized performance, and data protection.
