This is part three of a four-part blog series that will take a closer look at the visual effects (VFX) workflow. In our previous posts, we covered the components of the VFX workflow, as well as the rendering process. Today we’re discussing the common data-related challenges that studios and artists face with VFX workflows.
Complex Workflows, Complex Data
A visual effects (VFX) production pipeline is complex, with many processes.
One of biggest challenges is managing the sheer volume of data required to produce photorealistic imagery. A single digital creature might be composed of hundreds, if not thousands, of digital assets. It is often necessary to assemble terabytes of data that must be rendered and/or composited.
Volumetric data, which is essential to many bread-and-butter effects, including clouds, dust, water, fire, and fluids, is another example of extreme complexity. The data is challenging both because of its large footprint and because it often requires conversion to other formats before it can be used by other tools.
In general, the driver behind the growth of VFX is the fact that the technology has gotten better, cheaper, and faster. This means that animation and special effects have gotten increasingly ambitious. In addition, visual effects and animation have become critical during COVID to complete shows, commercials and movies that otherwise would have been shot on-set with film.
These complex effects require lots of data management and a data platform that is designed to handle billions of files. The emphasis is on “file” because VFX is a file-based workflow, and files are the medium of exchange between applications that were not necessarily written by the same company. Workflows must integrate across applications, and file data is the way to do that.
Data Pains: Performance, Scalability, Adaptability and Visibility
Any organization that relies on its file data platform as heavily as VFX studios do is always sensitive to a variety of issues. These include performance, scalability, visibility, and flexibility.
As more and more studios move to 8K and higher resolutions, performance will become increasingly critical. Systems that are too slow can starve the rendering farm or keep artists from working while rendering is going on.
Scaling Is another challenge. The ever-increasing amount of data that VFX shops generate can easily fill up their data storage system.
Problems such as scalability and performance, which you would expect from any organization that has to deal with an ever-growing number of files, are exacerbated for VFX studios because they cannot tolerate delays in their schedule to add more resources.
Depending on what file system they’re using, demands for more capacity may mean that the studio must replace the entire system or buy new metadata controllers and storage shells. With tight deadlines, there’s usually no time to build out the physical infrastructure.
Visibility is another limitation of legacy file systems. Most VFX shops use a treewalk script/program to manage their file data. Treewalks are extremely slow when file systems are large. Administrators can literally wait for days to get answers. By then, it’s too late for them to use the data in any way that matters.
Here’s a common scenario: A small VFX shop might, when it first starts, use a storage system the team has put together themselves. As their business grows, that system will need to be replaced. The artists will need a system that’s fast enough, so that they can work while frames are being rendered. They will need to move to a scalable, commercial file data platform with enterprise features will help them take on more projects and create more sophisticated effects.
The Challenges of Rendering in the Cloud
The ability to leverage the cloud is a major benefit for VFX studios, but cloud rendering is not without its own set of challenges.
Many of the same considerations for on-prem rendering apply to cloud rendering, but there are some specific issues that need to be considered.
Unfortunately, while compute resources in the cloud are readily available, traditional file-based storage solutions are often inadequate, or are versions of legacy file systems with some patches applied to make them “cloud-ready.” Problems include lack of protocol support, performance and capacity limitations, and complexity in setting up the cluster.
It’s important that studios match the performance of their on-prem render farm in the cloud. They need to be able to scale performance and capacity separately, to take advantage of the flexible resources the public cloud offers. Studios also need the ability to transfer files from the on-prem cluster to the instance in the cloud, and to then return the results back to the on-prem cluster.
Most of the file-based cloud systems cannot address all of these requirements. Qumulo’s file data platform was designed to solve all of the pains outlined above.
- Performance: Ultra-fast performance handles massive files with ease.
- Scalability: Simple modular design. Just add nodes to scale to petabyte levels. Gain performance and capacity in minutes. No disruption or downtime.
- Flexibility: Runs on-prem and/or in the cloud.
- Visibility: Monitor performance, capacity, and usage built-in real-time analytics.
Qumulo has several helpful resources for learning more about VFX and how our file data platform helps speed VFX production pipelines. Check out our data sheets on VFX, Animation, Media and Entertainment and learn how we’ve helped companies like FuseFX and Cinesite Studios to bring movies and animation programming to audiences faster!
Special thanks to Matt Ashton and Kristi Whitman for their contributions to this blog series. In the next blog, the last of this series, we’ll share examples of how VFX studios are innovating faster with Qumulo’s file data platform.
Linda is a Senior Solutions Product Marketing Manager at Qumulo, bringing over 20 years worldwide technology marketing experience with multiple industries, including enterprise software and hardware companies, SaaS, financial, healthcare, media and entertainment, research and manufacturing organizations.