Attributes of a Modern File Storage System

Posted January 10, 2018 by Stefan Radtke — Technical Director - EMEA.

This post features excerpts from a post originally appearing on Stefan's blog, which you can read here

About two decades ago, a number of parallel and distributed file systems were developed. The impetus was that, when data began growing exponentially, it became clear that scale-out storage was the paradigm to follow for large data sets. WAFL, IBM Spectrum Scale (aka GPFS), Lustre, ZFS and OneFS are all examples of scale-out file systems. All these systems have something in common: they had their "first boot" sometime around the year 2000. They also all have their strengths and weaknesses. Some of these systems are not really scale-out; others are difficult to install and operate; some require special hardware or don't support common NAS protocols; they may have scalability limits, or lack speed of innovation.

The fact that these systems were designed 20 years ago is a problem. Many important Internet technology trends such as DevOps, big data, converged infrastructure, containers, IoT or virtual everything were invented much later than 2000, so these file systems are now used in situations they were never designed to handle. It is clearly time for a new approach to file storage.

QF2 was designed by several of the same engineers who built and created scale-out storage roughly 15 years ago, and obviously their experiences led them to a very modern and flexible solution.

QF2 is hardware independent

Several vendors say their product is independent of hardware-specific requirements. They may have used the term "software defined." According to Wikipedia, two qualities of a software-defined product are:

It operates independent of any hardware-specific dependencies and is programmatically extensible.

QF2 fulfils both requirements admirably. You can run QF2 on standard hardware provided by Qumulo, on HPE Apollo 4200 servers, and in AWS. For development and testing purposes, Qumulo offers a free OVA package so that you to run a fully functional cluster on VMware Workstation or Fusion. You can also run a standalone instance of QF2, with 5TB of storage, on AWS for free. You only pay for the AWS infrastructure.

Because QF2 ist fully manageable via an API, it's fully extensible and it can be integrated into any operational environment.

Download the free OVA or set up an AWS instance to try QF2

QF2 runs in user space

The QF2 core OS is built on Ubuntu. Qumulo developers can leverage all the capabilities of the Linux ecosystem. The QF2 file system processes run in Linux user space rather than in kernel space, which has a number of advantages:

  • QF2 has its own implementations of protocols such as SMB, NFS and LDAP, which are independent of the underlying OS. For example, NFS runs as a service with its own notations of users/groups. This makes QF2 more portable.
  • Kernel mode is primarily for device drivers that work with specific hardware. By operating in user space, QF2 reinforces its hardware independence. It can run in a wide variety of configurations and environments
  • Running in user space means that Qumulo can develop and deliver features at a much faster pace.
  • Running in user space improves QF2 reliability. As an independent user-space process, QF2 is isolated from other system components that could introduce memory corruption, and the QF2 development processes can make use of advanced memory verification tools that allow memory-related coding errors to be detected prior to software release. By using a dual partition strategy for software upgrades, Qumulo can automatically update both the operating system and Qumulo Core software for fast and reliable upgrades. You can easily restart QF2 without having to reboot the OS, node or cluster.

Interactive API, real-time analytics and quotas

QF2 is programmatically extensible. It has a complete API, which can be extended and integrated into any datacenter environment.

If you like, you can use the API as the primary interface for all your management and operation tasks. However, for convenience, there is also a web UI and a CLI available. Both the UI and CLI use the same API that anyone can use to interact with QF2. The API and Python bindings are documented and available on GitHub. The same is true for the command line wrapper, qq.

One of the smartest things in QF2 is its real-time analytics capabilities. Metadata, such as bytes used and file counts, is aggregated when files and directories are created or modified, which means the information is available for timely processing without expensive file system tree walks.

The web UI includes a large number of real-time dash boards and graphs such as IO hotspots and throughput hotspots, and all the data can be retrieved via the well-documented API if you would also like to process it with other tools.

Stefan Radtke
Technical Director - EMEA
Qumulo is bringing long-awaited innovation to file storage. Most of the technologies dominating the enterprise today weren't yet invented when other vendors entered the market, and it is clearly time for a new, modern approach to file storage.

Stefan Radtke has spent his career working in technology, and comes to Qumulo as a principal evangelist of universal-scale storage for EMEA. Most recently, Stefan has been working to bring the best storage solutions to the automotive industry.

Let's start a conversation

We are always looking for new challenges in enterprise storage. Drop us a line and we will be in touch.

Contact Information



PR & Media:




Enter a search term below