Easily scalable infrastructure enables innovation for Atlantic
Atlantic specializes in providing remote sensing, surveying, and consulting services. Established in 2005, Atlantic offers an experienced staff of mapping and GIS professionals, many of whom are known as thought leaders in their field.
Further, through an investment in superior technology, combined with an in-house team of remote sensing and aerial mapping experts, Atlantic is uniquely positioned to support even the most challenging projects for its government and private sector clients.
“We’re growing rapidly and in this age of data explosion, data is key to everything. We’re a big part of that,” said CEO Brian Mayfield. “Our clients have a very diverse set of needs. They’re looking for solutions and geographic-enabled data that helps them solve problems. Those problems could be sea level-rise studies, flood map determination, forest inventories – we have a wide variety of needs we support.”
“Our vision is to be purposely different than any other company in the mapping profession by understanding our clients’ needs and innovating to those needs,” continued Mayfield. “As a business, we’ve been growing at about a 25 percent rate year-over-year. So in a four-year period we expect to double. Our data is growing at a much faster rate than that.”
In such a dynamic and rapidly growing field, the Atlantic team has to be ahead of the curve when it comes to innovation. “The technology during the course of my career has seen a substantial amount of growth. It seems like every year there are new sensors and techniques being developed,” said Mayfield.
“Our goal is to provide outstanding data to our clients. Our deliverable as a company is data. We deliver geospatial data, lidar data, photogrammetric data, and all of that has to be hosted somewhere,” said Ben Hayes, Senior Systems Architect at Atlantic. “Our job in IT is to make sure that the data is always available and that our storage is always scaling.“
“Other storage solutions require much more integration, much more set up, much more testing, much more everything, honestly. It was extremely different with Qumulo, for the positive.”
Innovative technologies and partnerships are driving massive increases in Atlantic’s data needs.
Cinesite knows first-hand the challenges of resource planning. “We can’t plan for the peaks, so we
Atlantic is a USGS Prime Contractor involved with the 3D Elevation Program (3DEP), which is a cooperative of agencies, all of which are committed to collecting 3D elevation data that helps scientists understand life-threatening natural hazards, and the impact of climate and land-use change.
“Working together, they’re able to buy large elevation data sets more economically than if they were working independently. For us it means we’re producing much larger areas than in the past. Our data needs and requirements are growing exponentially by the day,” explained Mayfield.
Another important driver of the company’s data requirements lies with the complexity of technology Atlantic is employing. “The technology that we’re most engaged with right now is called Lidar. It’s an active laser system that we operate from our manned aircraft,” said Mayfield. “The lidar puts down up to a million pulses per second of unique energy. Each one of those discrete pulses of energy returns a unique data point that’s stored in a data file, so those data points and record files get very large, very quickly.”
Lidar has been on the leading edge of technology in the geospatial and mapping industries for the last 15-20 years, and that will continue to be the case as the technology continues to mature and develop.
plan for an average,” said Peddie. That is the way planning had worked in the past. It was clear Cinesite would need a modern, cloud- native solution to move to a competitive scale.
“With our network having such a low latency backbone, and Qumulo having the ability to provide those high IOPS at lower latency, the software is able to react quicker to the requirements of the storage itself.”
Data-intensive workloads demand a stellar IT technology
Atlantic’s IT organization takes a proactive approach to supporting the needs of the growing company. Their focus is not just on managing technical challenges, but on finding innovative technologies that enhance the experience of their users.
“We focus on designing our infrastructure from the ground up, in addition to responding to user requests,” said Hayes. His approach to evaluating those technologies follows the same rigor. While most vendors offer minimum specifications, Hayes digs into the optimal performance of the systems he invests in, and builds Atlantic’s infrastructure around that.
Data storage is a primary example of this. With large file formats – such as incompressible, high-resolution aerial imaging data, the Atlantic team has very specific storage needs. “We need lots of space, high IOPS, low latency, and the ability to scale,” said Hayes. “When you’re looking at solutions for our requirements, there are only a few companies for us to consider.”
Mayfield is very tuned in to the importance of infrastructure technologies on their workflow. “We have to look at every single aspect of our IT infrastructure, from the backbone to our hardware solutions. That’s what we’ve been doing the last couple years, really evaluating where we have gaps, where we need to make adjustments, and where we can really gain a return on investment by investing in hardware and networking infrastructure,” he said.
“We’re generating data every day in the field. If we have every aircraft flying, that’s 20 TB of new data per day. When that data gets to the office, we have a short amount of time to determine if it’s quality data so that we can mobilize our aircraft somewhere else,” said Mayfield. “So getting data, loading it quickly, running highly interactive processes quickly, and getting results is key. Speed and size – being able to accommodate both of those variables is critical to our operation.”
These are the factors that led the Atlantic team to evaluate Qumulo. Mayfield remembered they were particularly impressed by the successes of Qumulo’s customers with high volume data needs, and by Qumulo’s unique customer support structure. “It piqued our interest and actually inspired some changes in the way we deal with our own clients.”
BENEFITS
- Peace of mind
- Long-term investment & customer partnership
- Cost-effectively scales to meet future capacity and performance requirements without breaking the bank
- Automation & control
Qumulo’s commitment to delighting customers was clear, and easy
The easy access to experts, and proactive communication from the Qumulo team was a big selling point for Atlantic and continues to impress them as they continue the partnership longer term.
“One of the exciting things about working with Qumulo for us is having the ability to chat directly with the engineering team on slack. You don’t have to go on with this long duration of ‘press one, press two,’ – you just open slack, post your question, and within seconds you could have two, three or 10 well-seasoned engineers responding to our questions,” said Mayfield. “And these are the people who develop the products, and support companies like ours.”
“You just don’t get that with other companies,” said Hayes. As the primary owner of the storage system, Hayes has had the most interaction with Qumulo’s success team, but not only if problems arise. Hayes has found that the Qumulo team offers him value through the Slack channel, even when it comes to casual questions, ideas, or requests.
A usable API offers a path to better control
“Qumulo provides awesome analytics, and all of those analytics are in real-time. But we required a little bit more in-depth analytics,” said Hayes. The Atlantic team was looking for more specific historical data than the real-time analytics or snapshots feature offered. In order to get the specific analytics Hayes was looking for, he looked to Qumulo’s API and customer success team to bridge the gaps he identified.
To start the conversation, Hayes headed to the Slack channel and started asking questions about how he might be able to get the functionality he was looking for.
“I had already accessed the API and was working with it,” said Hayes. Through the Slack channel, he was connected with Qumulo’s data visualization expert. Together, they figured out a way Hayes would be able to build his own dashboard utilizing the API that delivered the exact information he was looking for.
“That insight impacts us quite tremendously,” said Hayes. The added functionality allowed Hayes to look at any point in time to investigate activity within his file system. As problems arise, that visibility offers his team exactly what they need to keep operations moving smoothly and decrease bottlenecks.
“We just pop another node in the rack, press a button, and guess what? More space.”
A transparent system yields transparent problem solving
Working with the API isn’t the only way in which the visibility of the Qumulo system is impacting Atlantic’s operations. The heavily monitored system offers Hayes peace of mind.
“Qumulo helps take a lot of the stress that comes with storage management and giving me the peace of mind that we have a good solution in place that’s going to do what we want, and is heavily monitored not only by us, but by Qumulo support. And if there are any problems, Qumulo will respond extremely quickly and they’ll send somebody out if we need them to,” said Hayes.
The visibility also allows Hayes to have open conversations with his team about their activity within the system. “We have a standup meeting every Monday morning where I report on how much data has been added or removed and I will mention a few projects that are outliers,” said Hayes.
To accomplish this, Hayes uses the “capacity trends” dashboard. This allows him to pinpoint unexpected activity openly with the team so they can identify where problems are arising. If they see an unexpected spike or decline, they can discuss as a team to get to the root cause of the activity before it becomes a bigger issue.
Simplicity and scale
Working with the API isn’t the only way in which the visibility of the Qumulo system is impacting Atlantic’s operations. The heavily monitore
Since their implementation of Qumulo, the Atlantic team has been thoroughly impressed with the scale and performance of their system. The ease of use has also been a bright point in their experience.
“My personal experience has been fantastic,” said Hayes. “In the past, so much of our time revolved around managing storage and managing problems with the storage, whether it was issues from maxing out hard drive capabilities, or running out of space, or having to archive the data quicker.”
“Storage technology of the past is so radically different than the way Qumulo is,” continued Hayes. “There was no ‘stack and rack’ and turn on. No ‘press a button to set up a cluster’ and just go. That technology just didn’t exist. Other storage solutions require much more integration, much more set up, much more testing, much more everything, honestly. It was extremely different, for the positive.”
d system offers Hayes peace of mind.
“Qumulo helps take a lot of the stress that comes with storage management and giving me the peace of mind that we have a good solution in place that’s going to do what we want, and is heavily monitored not only by us, but by Qumulo support. And if there are any problems, Qumulo will respond extremely quickly and they’ll send somebody out if we need them to,” said Hayes.
The visibility also allows Hayes to have open conversations with his team about their activity within the system. “We have a standup meeting every Monday morning where I report on how much data has been added or removed and I will mention a few projects that are outliers,” said Hayes.
To accomplish this, Hayes uses the “capacity trends” dashboard. This allows him to pinpoint unexpected activity openly with the team so they can identify where problems are arising. If they see an unexpected spike or decline, they can discuss as a team to get to the root cause of the activity before it becomes a bigger issue.
Qumulo’s performance keeps up with the need
Performance is another key concern for the Atlantic team, and Qumulo has been holding up to the task. “Software’s not choking anymore, we’re able to keep data for slightly longer, we’re able to get projects out quicker. With our network being such a low latency backbone network, and Qumulo having the ability to provide those high IOPS at lower latency, the software is able to react quicker to the requirements of the storage itself.”
Part of these performance requirements are driven by the dramatic elasticity required by Atlantic’s workflow. “A single one of our projects, just the raw data could be 40TB,” explained Hayes. “We have three stages; first, the raw data comes from our sensors, then we have a working directory that is just the data that’s generated from the acquisition, to get to the final product. So it goes from 40TB to 140TB to maybe 15TB that we deliver to the client. We need to be able to get through that workflow and have the space available.”
“For our system to operate at what I’d call a neutral or an idle level, that is not choking on the data, we need to process between 20 and 40 thousand IOPS, that’s for a software not to choke on the data and needs to be provided at really low latency. So sub ten millisecond, essentially,” said Hayes. “We were seeing this ‘choking’ with our previous solution as we were maxing out the drive performance and the effects were very visible.”
But switching to Qumulo has completely alleviated this pain for the Atlantic team. “As far as the performance of our Qumulo cluster, 40,000 iops is roughly one fourth of its quoted capability. When we purchased it, it was sized for about 90,000 IOPS, and the cluster is just breezing along. It’s not even feeling it,” reported Hayes. “The cluster isn’t having any problems as far as performance goes. That’s a great thing, we have a lot of overhead we can play with if we need to add more machines, or pull the data in different directions.”
Mayfield has noted the improvements as well. “Our operational efficiency has improved overnight with the addition of our Qumulo device. Our processing rates have almost doubled, and we know Qumulo is having an impact on our performance increase.”
“Our operational efficiency has improved overnight with the addition of our Qumulo device. Our processing rates have almost doubled, and we know Qumulo is having an impact on our performance increase.”
The future of Atlantic’s partnership with Qumulo
Atlantic is anticipating exponential growth in the coming years. “Currently our storage solution has about a petabyte. I would say the ability to grow that to five petabytes in the next two years, or 10 petabytes in the next five years is a realistic opportunity for us,” said Mayfield. “The ability to really understand how to consistently scale our solution, and get the full ROI on the investment we’ve made so far is going to be the key to the relationship with Qumulo moving forward.”
As the company continues to scale its operations, and as the technology becomes increasingly advanced, their data storage needs will continue to increase. “Finding solutions that allow us to scale quickly without adding a lot more people are key. And Qumulo is certainly a part of our solution for that,” said Mayfield.
ABOUT
Established in 2005, Atlantic offers an experienced staff of mapping and GIS professionals, many of whom are known as thought leaders in their field. Atlantic specializes in providing remote sensing, surveying, and consulting services.
USE CASE
- Geospatial imaging
- Lidar
- Mapping
REQUIREMENTS
- Rapid growth
- Flexible scale for capacity & performance
- Ease of use