A Unified Data Platform for Foundation Model Training

Below are the key elements for the supporting foundation model training using data locally or data that is distributed geographically:

1. High-throughput, low-latency access to data for sustained GPU utilization locally and remotely.
2. Elastic scalability to handle multi-petabyte datasets and dynamically sized compute clusters.
3. Multiprotocol support to provide optimal file-based libraries like PyTorch
4. Global visibility and governance to meet data residency and compliance requirements.

Qumulo delivers each of these functions with its unique run-anywhere architecture that allows the unique features to manifest across any mix of compute, memory, and storage.  This enabled the data platform to operate with best-in-class costs and performance in the cloud (AWS, Azure, GCP, and OCI) as well as on any server hardware on-premises including from HPE, Cisco, Penguin and others.  This is the foundation of a unified platform, but just as important as the is the Qumulo multiprotocol support for applications to access and govern the data across these locations. This makes for easy integration across an enterprise application ecosystem, including various AI development toolchains like PyTorch, TensorFlow, and JAX.  This also is key for enabling easy integration with Amazon SageMaker Hyperpod.

Guaranteeing high performance is key in expensive training runs, where Qumulo NeuralCache functionality delivers optimal performance in local and geo-distributed data architectures. As an example, average latencies were sub-millisecond for training llama-3.1-8B using multiple AWS P5e EC2 instances (each with 8 Nvidia H200 Tensor Core GPUs).  We collected further performance data that will be posted separately for both single region and multi-region deployments.

For AI teams, developing models requires multiple training runs, which commonly use data outside of the site with accelerated compute resources.  An example of this is the need to run a training run in region 1, but being delayed by days to even weeks due to GPU scarcity issues. The do-it-yourself (DIY) approach results in hundreds of pipelines to stage each preselected dataset for training and validation runs at an accelerated compute site. This results in delaying time to train, data sprawl, lack of consistency of datasets, and governance issues. An alternative approach is for a unified data platform to move the data on-demand on your behalf to ensure performance, cost efficiency, and consistency amongst sites.  With Cloud Data Fabric (CDF), Qumulo provides a locality-aware data platform. CDF effectively extends a directory/prefix from a “hub” to one or multiple “spokes”.  Spokes are fully coherent and use Neural Cache’s locality awareness and model ensemble to decide which data to prefetch for each workload.  CDF  enables spokes to provide applications with local performance as a result.  We illustrate a verified architecture with SageMaker HyperPod and Qumulo in Figures 1 and 2. 

This is effectively a new data paradigm for AI teams that will reduce engineer time to results and increase performance of new models by virtue of an optimized zero-orchestration data loop.  Cost modeling shows that using this approach can drive down TCO by more than 30%. To achieve these goals, key features for a unified data platform with Qumulo are outlined below;

Elastic, High-Performance Data Access for Accelerated Compute

• Scale from 1 GB/s to 1 TB/s throughput with sub-millisecond latency as show in AI-Image benchmarks.
• Keep GPUs fully utilized by eliminating I/O stalls during data loading, sharding, checkpointing, or fine-tuning.
• Adapt automatically to each training workload using NeuralCache™ AI-driven caching and prefetching locally and remotely.

Unified Data Fabric for Workload Mobility

• Optimized latencies on spokes using NeuralCache™ AI-driven caching and prefetching seamlessly from the hub Qumulo clusters to/from multiple spoke Qumulo clusters.
• Compress, deduplicate, and WAN-optimize data transfers to cut data transferred by >30% as compared with manual copies.
• Apply consistent data across all environments with shared metadata, such as version information.

Multi-Protocol Flexibility

• Native support for S3, NFS, SMB, REST, and SFTP eliminates refactoring, re-architecting, or redundant copies..
• Data scientists, engineers, and simulation teams can access the same datasets with their preferred tools.
• Reduces data orchestration overhead by up to 4× versus traditional siloed storage.

SageMaker HyperPod with Qumulo

To demonstrate, Qumulo verified a common foundation-model workflow using SageMaker HyperPod deployed in the same region as the training data as well as deployed in a separate region from the training data set. The result is global model training without data duplication nor orchestration, but preserving consistency and reducing total cost of ownership.  One key detail is that, in most AI model development efforts, we see only 30% of the labeled data used per training run, and significant overlap across the first and subsequent runs.  This leads to higher ROI as compared to replicating your dataset per run, which is a common scenario for most teams in their orchestration efforts.  Even more impactful to improving ROI is that many teams have built complex data catalogues to avoid redundancies for this purpose, which is no longer required to build, maintain, and evolve when Qumulo CDF is leveraged.

SageMaker HyperPod with Qumulo – Single-Region DataSet

In this deployment, we deployed CNQ with the training data in the same region with SageMaker HyperPod.  CNQ was deployed in a single availability-zone approach that was not collocated with the P5e EC2 nodes.  The architecture is shown in Figure 1. 

SageMaker HyperPod with Qumulo – Multi-Region DataSet

In this deployment, we deployed CNQ in region 2 as the hub and region  1 as a spoke.  CDF would optimally distribute data to/from the CNQ Spoke folder.  This allows only the data needed for training to be distributed to the spoke.