EDITORIAL UPDATE:
- 9/23/2021: As we learn more we want to make sure our info is accurately represented. So, we have made adjustments to this story in the spirit of more accuracy.
- 9/15/2021: In the original LinkedIn post (posted 9/2/2021) I had made some assumptions about AWS FSx for NetApp ONTAP and how it seemed the solution may be configured from a storage perspective and some of its features. Thanks to our partner, AWS, those assumptions have been corrected as part of this update. I also took time to add more thoughts to this blog to be absolutely clear about the points that I am making.
File on AWS doesn’t get any better with NetApp
Today at AWS Storage Day the general availability of Amazon FSx for NetApp ONTAP was announced. AWS FSx is a managed service offered by AWS, and this makes the third such offering by AWS. It wasn’t clear to me at the time of launch what the solution was at face-value. However, after researching other blogs on this topic, and this one in particular on Judsonian.com, it seems this is Cloud Volumes Ontap (CVO). Now if you’re familiar with that, you’re familiar with the complexities that come with scaling. Qumulo scales very easily, without placing the onus on you, the end-user, to have to pre-plan or strategize the potential capacity required.
Throughput gets lost when you scale up
The announcement claims this solution is “virtually unlimited” in scale. However, based on what I read, there are two separate storage tiers. Bottom line, the maximum volume size for CVO is 100TB. If you want to scale beyond 100TB, you will need to do some pre-planning and strategizing because expanding it after you already have data will be a challenge. They also introduced an “intelligent tiering” feature with the offering. Sounds simple, but if you aren’t satisfied with its “auto tiering” default option you have to configure the what, when, and how for your data to be “intelligently” moved.
It also appears the larger volume that yields greater capacities, is an S3 bucket. Remember you do need to perform significant planning to strategize your needed capacity for the primary storage, and if you don’t then you may have to evacuate the data to tear down and rebuild the solution so you have the data spread out across all of these “flex vols.” So while FSx for NetApp ONTAP may scale to petabytes of storage, it isn’t as easy to do as it is with Qumulo. As for the “other tier” of storage or the “capacity tier,” it seems as though this solution is taking a page out of the “tape” manufacturer’s playbook and using S3 as a substitute for tape in order to “market” “virtually unlimited scale.” Technically you have petabytes of data, but the real question is how quickly the data can be accessed and put to use. Let’s be frank, petabyte scale to me means, petabyte scale in a single namespace or volume and in a reasonable state where you can easily and quickly consume it. If you don’t size your primary tier properly, you have the safety net of the “capacity tier,” but at that point it is more of an archive than the single namespace intelligent storage solution Qumulo offers its customers.
In the event you need that data ASAP, will you have any guarantees, besides uptime, on the performance of that data retrieval? If it is true that the capacity pool is S3, then I assume the SLA I found when I searched for “AWS S3 SLA” is what applies to this solution for customers of FSx for NetApp ONTAP. When I read this, I don’t see anything around performance guarantees, only uptime. It would appear any performance guarantee from the capacity pool to the primary volume or namespace is unknown, as far as I can tell. Where Qumulo differs is how we intelligently tier the active and inactive data without forcing our customers to compromise on performance, access or the ability to scale capacity. We don’t believe tiering should be difficult, in fact, it shouldn’t be on your shoulders at all, but in the hands of an intelligent filesystem. Qumulo has long been a leader in intelligent tiering, in fact we have been delivering solutions with both SSDs (for read/writes) and HDDs for years. In a blog post by Qumulo’s Product Manager, Jason Sturgeon, Jason points out “Qumulo’s machine learning (ML) cache leverages both pattern matching to read ahead data, as well as block level activity for heat analysis. The intelligence actually looks at not only the clients, but the sessions on that client to determine what caching mechanisms will work best. The machine learning cache enables over 87% of data to be read from memory or NVMe SSDs in a hybrid environment.” So, as you can see, with Qumulo there is no guesswork and the yield for you is better performance access to your data without question.
It also seems to be on you, the customer, to determine what your file efficiency will be with deduplication, compression, and compaction. You’ll probably hear your NetApp or AWS FSx rep tell you, “your mileage may vary” when you ask for help. That’s the reality with tech such as that, it really is a guessing game and if you’re off, then more than likely your pricing will go up. Amazon FSx for NetApp ONTAP is complex at its best. The Qumulo filesystem is much more efficient than ONTAP, therefore we do not need the heavy overhead of deduplication/compaction/compression. Qumulo is truly a modern file data platform born in this century and designed with cloud-first in mind. This is why customers will find storing the same amount of data on Qumulo can take up to 25% less raw capacity than the competitors. This is accomplished by leveraging erasure coding (EC) as opposed to mirroring. This boils down to one very important element, data protection. Protecting your data should be top priority for any storage manufacturer and it certainly was with ONTAP when it first came out in the late 90s, but those methods are rudimentary at best and grossly inefficient for today’s modern solutions. We found that not only can we be more efficient with EC, but it also allows our customers to consume 100% of their usable capacity. To learn more about how Qumulo is different from the legacy competitors, please read this white paper for more information.
Intelligent tiering? A copout for complexifying the cloud
The open question I still have about this solution, even given the “intelligent tiering,” is if NetApp will still strongly urge its customers not to exceed 80% of the primary storage. I mean, it would make sense since it is ONTAP and ONTAP doesn’t like it when it gets too full. Will performance be gated, will the tiering criteria be overridden and create a mass data migration from the primary to the secondary to save space and avoid a performance nightmare? Unknown right now, but knowing what I know about ONTAP, it wouldn’t surprise me to see this happen.
We all know NetApp does have performance issues, that’s an inherent problem with all scale-up solutions. That’s why you want a scale-out, distributed file system like Qumulo. We are achieving up to 40GB/sec performance compared to FSx for NetApp ONTAP performance of 2GB/sec of read performance and 1GB/sec of write performance. So now I’m more curious. Does the 2GB/sec read include the read from the capacity pool or is it only from the primary storage volume on FSx for NetApp ONTAP? Unclear, I can’t find any reference other than what I mentioned above. You know I had to dig a little to find the actual performance numbers. It was buried in the FAQ.
Adding up costs at petabyte-scale
NetApp uses a 100TB example in their announcement but when you look at a petabyte scale use case with data that you need to actually use for workloads such as AI model training, genomic sequencing or video production the cost model breaks the bank. Alternatively, with Qumulo you really do have the ability to scale without having to pre-plan or estimate where your capacity may need to be for your particular workload. If you need more capacity, even petabytes of capacity, you simply add more capacity and the namespace grows seamlessly. I think the best part about Qumulo is there is nothing else left to purchase. When you buy Qumulo Core all of these features are included.
- Qumulo Core Includes:
- Qumulo Scale
- Dynamically scale to store petabytes in a single namespace
- Qumulo Perform
- Intelligent caching to meet your most rigorous workloads
- Qumulo Aware
- Instant visibility with real-time analytics
- Qumulo Integrate
- Automate workflows and build applications with our advanced API
- Qumulo Shift
- Easily move data to the cloud
- Qumulo Protect
- Easily protect your data to minimize disruptions
- Qumulo Secure
- Automatically protect data from external threat
- Qumulo Scale
Learn more about our data services here.
Here we took the exact calculation NetApp published and scaled it to a petabyte (which you can’t actually do in NetApp unless you like managing multiple volumes). It runs you at $72TB per month, and if you need to frequently access that data (let’s assume 80% of the time) the price shoots up to $142TB per month. When that same use case on Qumulo on AWS can run you at $52 TB per month.
If there’s any indication of complexity, just look at the number of “knobs and dials” you have to configure, each racking up the cost of your storage like a lottery ticket!
- Start with your volume size.
- Next, create a tiering profile by guessing how much data will be in your primary tier (frequently accessed data). That’s a charge.
- Now you have to decide how big your “capacity pool” is (infrequently accessed data). That’s another charge.
- Then you have to scope your throughput requirements. Yup! You guessed it…another charge.
- What about backups? That’s another dial you’ll have to hone in, and add an additional charge.
You shouldn’t need to make a complex set of decisions just to get your storage up and running.
With Qumulo you don’t need an abacus to decide any of that. Pick your capacity, your performance, and pump in as much data in as you’d like. And, it doesn’t matter how often or how little you use your data. Qumulo automatically tiers your data with intelligent caching, no more guessing games. And when we say “intelligent caching” it doesn’t mean you have to turn dials and tweak configurations. Qumulo has it covered for you, intelligently, within a single namespace capable of scaling to several petabytes of storage. That’s it. It really is that simple. No knobs, dials, or management headaches with hidden costs. Qumulo is a software platform with an all-inclusive pricing that won’t break the bank.
The net on NetApp
The need for file services in the cloud is growing exponentially and many customers are seeking out solutions that will meet their needs and an ability to “get out of the data center business” while focusing on the core of their own business. The problem is that legacy dinosaurs like NetApp are just bringing their outdated and complex architecture to the cloud and not able to deliver on the golden trifecta of scale, performance, and cost. Instead, customers should consider cloud native platforms that have proven scalability and performance and can deliver cost-efficiency at scale. Yes, I’m talking about Qumulo.
Qumulo can help you build petabyte-scale file environments that are price and performance optimized for your specific use case.
Click here to arrange a meeting.
This post appeared earlier on LinkedIn.
Contact us
- Take a test drive in our interactive hands-on labs, or book a demo.
- Subscribe to the Qumulo blog for customer stories, technical insights, and product news.
