This post features excerpts from a post originally appearing on Stefan's blog, which you can read here
About two decades ago, a number of parallel and distributed file systems were developed. The impetus was that, when data began growing exponentially, it became clear that scale-out storage was the paradigm to follow for large data sets. WAFL, IBM Spectrum Scale (aka GPFS), Lustre, ZFS and OneFS are all examples of scale-out file systems. All these systems have something in common: they had their "first boot" sometime around the year 2000. They also all have their strengths and weaknesses. Some of these systems are not really scale-out; others are difficult to install and operate; some require special hardware or don't support common NAS protocols; they may have scalability limits, or lack speed of innovation.
The fact that these systems were designed 20 years ago is a problem. Many important Internet technology trends such as DevOps, big data, converged infrastructure, containers, IoT or virtual everything were invented much later than 2000, so these file systems are now used in situations they were never designed to handle. It is clearly time for a new approach to file storage.
QF2 was designed by several of the same engineers who built and created scale-out storage roughly 15 years ago, and obviously their experiences led them to a very modern and flexible solution.
Several vendors say their product is independent of hardware-specific requirements. They may have used the term "software defined." According to Wikipedia, two qualities of a software-defined product are:
It operates independent of any hardware-specific dependencies and is programmatically extensible.
QF2 fulfils both requirements admirably. You can run QF2 on standard hardware provided by Qumulo, on HPE Apollo 4200 servers, and in AWS. For development and testing purposes, Qumulo offers a free OVA package so that you to run a fully functional cluster on VMware Workstation or Fusion. You can also run a standalone instance of QF2, with 5TB of storage, on AWS for free. You only pay for the AWS infrastructure.
Because QF2 ist fully manageable via an API, it's fully extensible and it can be integrated into any operational environment.
Download the free OVA or set up an AWS instance to try QF2
The QF2 core OS is built on Ubuntu. Qumulo developers can leverage all the capabilities of the Linux ecosystem. The QF2 file system processes run in Linux user space rather than in kernel space, which has a number of advantages:
QF2 is programmatically extensible. It has a complete API, which can be extended and integrated into any datacenter environment.
If you like, you can use the API as the primary interface for all your management and operation tasks. However, for convenience, there is also a web UI and a CLI available. Both the UI and CLI use the same API that anyone can use to interact with QF2. The API and Python bindings are documented and available on GitHub. The same is true for the command line wrapper, qq.
One of the smartest things in QF2 is its real-time analytics capabilities. Metadata, such as bytes used and file counts, is aggregated when files and directories are created or modified, which means the information is available for timely processing without expensive file system tree walks.
The web UI includes a large number of real-time dash boards and graphs such as IO hotspots and throughput hotspots, and all the data can be retrieved via the well-documented API if you would also like to process it with other tools.
Stefan Radtke has spent his career working in technology, and comes to Qumulo as a principal evangelist of universal-scale storage for EMEA. Most recently, Stefan has been working to bring the best storage solutions to the automotive industry.
We are always looking for new challenges in enterprise storage. Drop us a line and we will be in touch.
Enter a search term below