Most modern innovations and services—the ones advancing the human condition and creating a better world for ourselves and our children—are digital. They begin, evolve, and end with raw data. Gene mapping has been used for vaccine creation and the mapped gene data is stored in unstructured files. Personal movies from mobile phones and security footage from cameras are increasingly being shot in high quality 8K, the same quality as the latest blockbuster movie release. And those video files are stored in unstructured file format. Data sets used to train machines to do everything from automatically driving a car to determining the right place to drill for oil leverage vast amounts of unstructured data as part of their training sets. Anywhere you look, unstructured data is powering innovation.
When well-managed and then transformed, this unstructured data can be critical in creating our modern world. But most modern data technology wasn’t built to leverage it. Not only is unstructured data profoundly underutilized, it also faces its fair share of challenges. But the modern enterprises surmounting them are not only innovating and creating amazing things to make our lives better, they’re saving money and time in the process.
We’re living in a moment when there has never been more data.
Not all data is the same
When people think about data, they usually think about structured data. But in reality, customers, clients, and citizens generate far more unstructured data.
Both structured and unstructured data are invaluable, but they’re decidedly distinct. According to Fintech Futures, unstructured data accounts for approximately 80% of the data banks hold. This includes data stored as audio, video and email files, all of which are unstructured data files. Yet when it comes to unlocking the value of unstructured data, “very few firms are utilizing the information they gather,” said Ryan Stewart, writing for Fintech Futures. “The biggest barrier for the banking sector is its large-scale and outdated IT infrastructure, with 92% of the world’s top 100 banks still relying on legacy systems.”
Structured data vs unstructured data
Structured data is neat, tidy, and relatively easy to analyze. It can be easily stored in rows, columns, tables, spreadsheets, and databases. Nearly all data technology has been created in the last 10 years to manage and manipulate it. Unstructured data is its eccentric and unruly cousin.
Unstructured data which is natively a file format, so also referred to as file data—comprises 80% of all enterprise data. It includes image, audio, text, and video files—think emails, podcasts, social media posts, presentations, movies, medical imaging, genomic research, and more. Although unstructured data rarely fits neatly into standard boxes, it’s the substance of global change, innovation, collaboration, and transformation. And most of the opportunity and possibility with data lies in unstructured data. It’s time to pay attention.
Unstructured data drives innovation and transformation
Across industries, unstructured data is on the rise. According to leading analyst firms, enterprises will triple unstructured data stored on-premises, at the edge, or in the cloud by 2024. And in the wake of a global pandemic—as remote work has become commonplace—the cloud is no longer optional. Rather, it is essential for competitive advantage.
Unstructured data accelerates digital transformation. But to make new medicines, treat diseases, entertain ourselves, and develop intelligent machines that enable us to work faster, smarter, and more sustainably, we must not only collect unstructured data, but also transform it into something usable and useful.
Dayton Children’s Hospital, for example, leverages unstructured data to improve patient outcomes and save lives. Physicians at this top-rated teaching hospital depend on fast retrieval and secure archiving of high-resolution medical images for diagnoses and care at their level-one Pediatric Trauma Center.
Hyundai MOBIS, one of the world’s largest suppliers of car parts and components, is using massive unstructured data sets to develop training scenarios for its autonomous driving and connected car technology. This South Korean enterprise stores and analyzes hundreds of terabytes of video data to help make vehicles intelligent.
Industrial Brothers, a full-service animation studio—which lacked a cloud presence and didn’t support remote work prior to March 2020—leverages unstructured data to create, produce, and collaborate on children’s shows. When their central office was forced to shut down in response to COVID-19, like many organizations, they needed to pivot quickly. They virtualized their collaborative studio experience and migrated all their creative and production workloads to the cloud.
These are just three of the countless companies doing great things with unstructured data. They’re leveraging it to generate insights, improve business practices, inform decision-making, and drive innovation. But unstructured data must be well-managed and readily accessible to accomplish this type of work.
The usage and management of unstructured data is in its infancy. And as countless other organizations that manage and store data with outdated systems have discovered, data transformation is easier said than done.
Why unstructured data is a big problem
There’s no doubt, unstructured data is packed with possibility. But for many organizations, it can be—or become—a major problem. Here are seven of the most common reasons.
1. Organizations struggle to keep up with, manage, and access sufficient storage.
Raw data—often captured from sensors, cameras, sequencers, cars, or other machines—is of little consequence until it’s learned from and then transformed. This conversion of data to insights to innovation often requires collaboration across massive amounts of data. And data innovation requires data accessibility. Organizations often accumulate hundreds of terabytes or even a petabyte of data that they must store indefinitely. This is the storage equivalent of 1000 laptops! As data grows, so too must storage. Tons of data require tons of storage.
2. Legacy systems weren’t designed for modern workloads or the cloud.
The old guard of scale-out and scale-up solutions wasn’t designed to handle today’s applications, file types, workloads, and volume. And of the two primary ways to store and manage unstructured data—object and file storage—only file systems are designed to manipulate data in its native file format. Legacy and object-storage systems can’t provide the performance, visibility, portability, control, or scalability that modern data management and cloud migration require.
3. Legacy architecture limits scalability.
Legacy architecture is often on-premises and hardware bound. So, storage is subject to the magnitude of one data-center architecture. As compute scales, storage must also. But data center real estate is expensive. These limits can stifle creativity and exploration of new ways to build with unstructured data.
4. Data silos inhibit access and collaboration.
To deal with scalability issues, some organizations have turned to storage arrays or multiple data centers. While these solutions temporarily address storage issues, data silos and disparate storage arrays make real-time access and collaboration difficult. To optimize data insights and make them useful, consolidated data is ideal.
5. Consolidated data limits storage options.
Unfortunately, consolidated data also has limitations. It requires a bucket large enough to contain it, plus a scale sufficient for lots of users to transform it. Neither data centers nor public cloud offer more than a handful of storage options—and these limited choices aren’t great ones. An investment in bespoke data center hardware requires ongoing investments into more bespoke hardware. And if you’re locked into a data center, you’re locked out of the cloud unless you move to a hybrid cloud environment. Public cloud options that confine you to a specific cloud will also confine your compute, networking, and workflows.
6. Competitors are migrating to the cloud.
By 2022, leading analysts predict that public cloud services will be essential for 90% of data and analytics innovations. And forward thinking companies—and competitors—know this. They’re moving workflows to the public cloud. And unstructured data is only accelerating this migration. The faster organizations get to the public cloud, the more competitive advantage they gain.
7. Top talent is moving to modern workplaces conducive to remote work and collaboration.
Home-based workers lack sufficient infrastructure to be productive with large-scale data. They must go to the office to complete their work. But this won’t last long. Top talent will eventually choose cloud-based workplaces conducive to remote work and collaboration.
Do good work with unstructured data.
Managing, storing, and transforming unstructured data at scale to drive innovation may seem daunting. But as we embrace new business models, demand data platforms that offer freedom, control, and real-time visibility, and simplify the way we manage and store data, it’s both doable and possible.
Like other modern innovators, you can leverage unstructured data to do good work in the world. As you’re considering and reconsidering your own unstructured data and infrastructure strategies, here are a few suggestions.
1. Be humble about the future.
The cloud wasn’t a mandate three years ago, and now it is. When it became non-negotiable, we were all saying everything had to go to the cloud, but options were limited. Today, with AWS, Azure, and the Google Cloud Platform, options are plentiful, and choice has become a consideration. But what’s working today may not be tomorrow. So, have some humility about the future as you make decisions. Select infrastructure strategies that offer future flexibility.
2. Be intentional about what you lock into.
Be laser focused and selective as you lock into your strategies. Lock into applications that create value for your end users. Lock into infrastructure software that allows you to standardize practices and reduce complexity. Choose a stable file-data platform, which deals with unstructured data in its native file format. Choose flexible, cost-effective storage that transcends hardware, data center, and cloud limitations. And be skeptical of vendors and platforms with solutions that hijack this flexibility.
3. Be strategic in your move to the cloud.
As you move your enterprise to the cloud, remember this three-step framework: Consolidate, extend, transform.
- Consolidate your unstructured data and workloads in one place. This will reduce the costs and complexity of managing multiple systems.
- Extend your unstructured data and infrastructure into the public cloud. You can do this through cloud bursts or by building individual workloads that can vacillate between on-premises and the cloud.
- Transform workflows to be entirely cloud based. Sustainable digital transformation takes time. So, be patient, take strategic steps, and be mindful not to jump straight to transformation.
Business leaders willing to be humble and intentional about their infrastructure strategies, and who take strategic action to move to the cloud, can save time and money and retain top talent. With the right data platform, they can gain full control of their data and leverage the value and freedom of unstructured data to drive innovation.
What to Consider When Evaluating Enterprise Data Storage Solutions
Qumulo DataBytes: 43% of Execs say IT is a Business Inhibitor Due to Difficulty Accessing Data
