Migration to Azure Native Qumulo Scalable File Service with Atempo Miria

File migration between different storage platforms and file system architectures can be a complex process. This article describes a solution that uses Atempo Miria to migrate file data from third-party NAS systems to Azure Native Qumulo (ANQ). Atempo Miria provides platform-agnostic file migration services at massive scale in heterogeneous NAS environments. This solution facilitates the seamless migration of file data between mixed storage platforms, providing high-performance data movement, ensuring data integrity, and empowering you to successfully complete data migration projects from your legacy NAS to an ANQ instance.

Benefits of Migration to ANQ with Atempo Miria

Scalability	A single ANQ instance can scale to exabyte size and beyond in a single namespace. Miria is designed for high-performance data movement at limitless scale, simplifying the process of migrating billions of files from any legacy NAS system to ANQ.
Cost Efficiency	ANQ’s overall TCO is comparable to on-prem file services, even at scale. Customers pay only for the capacity and throughput they use while they use it. With its intelligent multithreading and parallelization features, Miria for Migration makes file migration smoother and faster, eliminating long migration cycles and optimizing your costs. Adding or removing Data Movers doesn’t generate additional costs for the Miria licenses.
Performance	ANQ outperforms other cloud-based file platforms, delivering higher throughput and lower latency for most workloads. ANQ also supports temporarily increased throughput as needed for burst processing. Superior performance with Fastscan integration on leading storage vendors to accelerate migration to ANQ from Isilon / PowerScale, GPFS, etc. Increase or lower overall throughput by simply adding or removing Data Movers to the solution. FastScan enables the rapid detection of files created, modified, or deleted since the last migration cycle.
Global Reach	ANQ can be deployed in one or more Azure regions worldwide. Miria Data Movers can be deployed in your data center and into the same Azure region as your ANQ for better performance.
Security and Compliance	Miria Migration collects files with their assigned access permissions and ensures full data and access integrity on the target storage and protocol.
Management Simplicity	Detailed reports at each step of the process enable the calculation of the migration convergence point, indicating how much data remains to migrate and any issues to resolve. The migration process can be transformed into data protection for the new storage using the same solution.

Architecture

The solution is deployed on Azure with configurable performance options and combines Azure Native Qumulo’s exabyte-scale file data platform and Atempo Miria, as shown in the following diagram.

Solution Architecture

This solution is deployed into an Azure customer tenant in a single Azure region with customer resources, including a virtual network gateway.
Atempo Miria offers several key features, including optimized data transfer, parallelization, multithreading, FastScan technology for quick object modification identification, automatic file migrations, iterative migration cycles, integrity checks, and performance adjustments. It also maintains data reliability by constantly comparing source and target storage, as well as the ability to use a hashing mechanism during synchronization.
The Miria Server is the central management component for the data migration process.
Atempo Miria Data Movers manage the data movement and are key components for the global performance of the migration operation. One set of Data Movers is deployed next to the existing legacy NAS system. A second set of Data Movers are in the customer’s Azure tenant for optimal performance, allowing them to pre-process and post-process jobs without any latency issues.
The ANQ cluster used in the solution is deployed in Qumulo’s Azure tenant.
Access to the ANQ cluster is enabled via VNet injection from a dedicated subnet in the customer’s Azure tenant, which connects to the customer’s dedicated ANQ cluster in the Qumulo tenant. All data on the ANQ cluster is accessible only via the network interfaces in the customer’s delegated subnet. Note: Qumulo has no access to any data on any ANQ instance.

Click here to download a high-resolution Visio diagram of the solution.

Solution Workflow

A migration project consists of 3 steps:

Step 1: First Synchronization: set off the initial data transfer and perform a first snapshot of the source. The snapshot is used as a reference point in time. The transfer then copies the source storage data to the target storage.

Step 2: Incremental Synchronization: propagates any modifications made to the source storage (since the previous incremental sync) to the target storage.

Step 3: Final Synchronization (Cutover): complete the data migration to the target storage while production access to data on the source NAS platform is halted.

Optional Step 4: Scheduled Synchronizations: If legacy NAS systems remain in use for specific on-prem data center applications, it’s essential to ensure data consistency between the legacy NAS system and ANQ. Miria’s Synchronization feature can be used to ensure that any data changes on one system are synchronized to the other.

The Miria Server manages the deployed Miria Data Movers. NFS workloads and SMB workloads require different OS deployments for the Data Movers. The Miria Server itself can be deployed on Linux or Windows OS.
Data Movers can be combined within Linux Pools or Windows Pools to add redundancy (in case of a failure on one Data Mover), increase overall throughput, or shorten the migration time for large datasets.
Access between the on-prem Data Movers and the Azure-based environment can be via Azure VPN Gateway or through an ExpressRoute connection.
The Atempo Miria service should run using an administrator or root-level user account on both the legacy NAS system and the ANQ instance to ensure full access to all files.
API connections to legacy NAS and ANQ storage are required to enable support for FastScan, SnapStor, and integrated snapshot management options. () Advanced storage integration allows the replication of the NFS Exports or SMB Shares and their respective configuration settings from the legacy NAS to the ANQ cluster with the Synchronization task. Besides Qumulo, FastScan features are available when migrating from Isilon, OceanStor, Nutanix NAS and GPFS shared file systems.
If the legacy NAS system includes both SMB and NFS data, it is possible to migrate datasets that include NTFS and POSIX permissions. Use Data Movers from a Windows pool when moving SMB data. For source files shared via NFS, the Data Movers should be selected from a Linux pool. Cross-protocol permissions, in which a single dataset is shared via both POSIX and ACL permissions, may require an additional step to migrate successfully. The initial synchronization (and migration of the data’s POSIX permissions structure) can be performed using Data Movers in a Linux pool. Due to the lower overhead of the NFS protocol, the NFS-first approach is usually the quicker method. Subsequently, an additional task can be created on the Data Movers in a Windows pool to migrate the NTFS permissions for the same data.
Select one or multiple source directories from legacy NAS and designate their targets on ANQ. Be careful to exclude directory definitions for the .snapshot and .snapshots directories.
Synchronization Type – Echo applies the synchronization from the source directory to the destination directory.
Parallelization shortens the migration window by creating multiple concurrent streams to migrate the data. A new job will start each time a threshold is reached during the scanning process. A threshold can be a time limit, a file selection size, a number of files, or a combination of these.
Select the correct hashing algorithm for integrity checks according to performance and collision prevention needs.
Test client connectivity to the SMB shares or/and NFS exports on ANQ.
Tasks are automatic jobs that you can schedule or start manually. A task defines the scope of a Miria job, the source and destination of the data that it processes, and its scheduling. The schedule defines the times at which automatic tasks kick off between the legacy NAS system and ANQ.

Components

The solution architecture comprises the following components:

Azure Native Qumulo Scalable File Service cluster to provide consolidated data storage for all content created using the solution
Atempo Miria to migrate data from local storage to ANQ.
- Miria Service
- Miria Data Movers (Agent)(minimum 16 core CPU, 32 GB memory, and 10 GbE network)
Azure ExpressRoute and/or Azure VPN Gateway
Azure VNet Injection
Azure Virtual Network
Legacy NAS (e.g. on-premises Dell EMC Isilon / PowerScale)

Considerations

File migration to Azure Native Qumulo Scalable File Service with Atempo Miria should include the following considerations in their planning and design processes.

Potential Use Cases

Data Migration: Cloud migration is the process of moving applications and data from on-premises systems to a public cloud. However, legacy NAS options in the market have limited functionality or scalability on the public cloud. ANQ offers enhanced functionality and scalability on Azure. Atempo Miria facilitates the high-performance migration of your file data to ANQ while ensuring data integrity.
Disaster Recovery: ANQ can be utilized as a secondary DR site, allowing organizations to replicate their data to the cloud. In the event of a disaster that impacts the primary on-premises site, organizations can switch to the replicated data on ANQ and continue operations with minimal disruption. Atempo Miria FastScan feature quickly detects files to be replicated from a source legacy NAS.

Scalability and Performance

When planning a data migration solution using Qumulo and Atempo Miria, enterprise architects and other stakeholders may want to include the following in the solution:

Capacity and growth – ANQ scales easily on demand, allowing you to add as much capacity as needed simply by creating or migrating data.
Atempo tools allow you to migrate file, folder, and ACL data from legacy NAS to ANQ with full automation, load balancing, and macro and micro-level reporting. We can scale the power of our tools thanks to multi-streaming, multi-threading Atempo Miria Data Movers.
Throughput – ANQ allows you to adjust throughput on demand in 1GB/s increments to ensure the availability of throughput you need at all times.
With the Miria software, the transfer rates between the source and target storage can approach the end-to-end connection’s network speed. Performance can be impacted by data type and layout: a large number of small files or a high directory count within a given dataset can slow down overall throughput by increasing the load on the Data Mover.

Security

The Azure Native Qumulo Scalable File Service connects to your Azure environment using VNet Injection, which is fully routable, secure, and visible only to your resources. No IP space coordination between your environment and the ANQ cluster is required.

Care should be taken during design and implementation to ensure that the solution’s security complies with industry best practices, internal enterprise policies, and any applicable legal/regulatory requirements.

For all other aspects of the solution, customers are responsible for planning, implementing, and maintaining the security of the solution to satisfy all applicable legal and regulatory requirements for their industry and location.

Cost Optimization

Cost optimization refers to minimizing unnecessary expenses while maximizing the value of the actual costs incurred by the solution. For more information, visit the Overview of the Cost Optimization Pillar page.

Azure’s pay-as-you-go model allows you to optimize costs by scaling resources to use the capacity when needed. This helps you manage costs efficiently, without over-provisioning resources.
The cost of the Qumulo depends on the amount of data on the Azure Native Qumulo Scalable File Service and the performance tier. For details, see Azure Native Qumulo Scalable File Services pricing.
Miria’s architecture is designed for scalability and to provide parallelized data flows. When you need more power and higher performance, you just need to add another Data Mover to match your needs.

Availability

Different organizations can have different availability and recoverability requirements even for the same application. The term availability refers to the solution’s ability to continuously deliver the service at the level of performance for which it was built.

Data and storage availability

The ANQ deployment includes built-in redundancy at the data level to ensure data availability against failure of the underlying hardware. To protect the data against accidental deletion, corruption, malware, or other cyberattack, ANQ includes the ability to take snapshots at any level within the file system to create point-in-time, read-only copies of your data.

ANQ supports replication of the data to a secondary Qumulo storage instance, which can be hosted in Azure, in another cloud, or on-premises. ANQ is compatible with file-based backup solutions to enable external data protection.

Service stability and continuity

Depending on the scope and criticality of the data being migrated, a phased approach may be appropriate in some circumstances. Rather than migrate the entire contents of your legacy storage in a single operation, a phased approach would entail migrating one workflow at a time, using smaller, less-critical datasets to develop and finalize the process before moving to larger and more-important data.

An added benefit of this approach is that any issues that may arise from a migration operation are limited in scope to a single workflow or dataset.