Benefits of Migration to ANQ with Komprise

Solution Architecture
Solution Workflow
Components

Considerations

Potential Use Cases
Scalability and Performance
Security
Cost Optimization
Availability
- Data and storage availability
- Service stability and continuity

Migration to Azure Native Qumulo Scalable File Service with Komprise

File migration between different storage platforms and file system architectures can be a complex process. This article describes a solution that uses Komprise to migrate file data from third-party NAS systems to Azure Native Qumulo (ANQ). Komprise provides platform-agnostic file migration services at massive scale in heterogeneous NAS environments. This solution facilitates the seamless migration of file data between mixed storage platforms, providing high-performance data movement, ensuring data integrity, and empowering you to successfully complete data migration projects from your legacy NAS to an ANQ instance.

Benefits of Migration to ANQ with Komprise

Scalability	A single ANQ instance can scale to exabyte size and beyond in a single namespace. Komprise is designed for high-performance data movement at a limitless scale, simplifying the process of migrating billions of files from any legacy NAS system to ANQ. Komprise’s data analysis, smart data migration, data tiering, and archiving capabilities are tailored for large-scale operations, making it ideal for any organization’s cloud strategy.
Cost Efficiency	ANQ’s overall TCO is comparable to on-prem file services, even at scale. Customers pay only for the capacity and throughput they use while they use it. Komprise smart data migration strategies lead to savings of over 70% by right-placing data across file storage, ensuring cost-effective migrations. With its parallelization features, Komprise for Migration makes file migration smoother and faster, eliminating long migration cycles and optimizing your costs. Adding, or removing, Proxies and Observers doesn’t generate additional costs for the Komprise licenses.
Performance	ANQ outperforms other cloud-based file platforms, delivering higher throughput and lower latency for most workloads. ANQ also supports temporarily increased throughput as needed for burst processing. Komprise offers faster NFS and SMB migration speeds, drastically outperforming conventional tools and methods. Utilizing Komprise’s patent-pending Hypertransfer technology, data is migrated 25x to 27x faster, ensuring high performance even in extensive migrations.
Global Reach	ANQ can be deployed in one or more Azure regions worldwide. Komprise Observers and Proxies can be deployed in your data center and into the same Azure region as your ANQ for better performance.
Security and Compliance	Komprise ensures that all file permissions and access controls are included during migration, maintaining data integrity and security. Chain-of-custody and MD5 checksum reporting at the file level are integral to Komprise’s approach, and essential for regulated industries.
Management Simplicity	Komprise allows for comprehensive analytics before migration, aiding in planning by understanding data by age, size, usage, costs, ownership, and helps to identify potential issues before starting a migration helping to ensure a fast successful move. For Cloud, and WAN, migrations, Komprise includes a non-intrusive assessment of the network environment, helping customers identifying potential bottlenecks, routing troubles and other issues that may be hidden until the heavy load of a migration occurs. This assessment before starting to move data helps ensuring smooth migration.

Architecture

The solution is deployed on Azure with configurable performance options and combines Azure Native Qumulo’s exabyte-scale file data platform and Komprise, as shown in the following diagram.

Click here to download a high-resolution Visio diagram of the solution.

Solution Architecture

This solution is deployed into an Azure customer tenant in a single Azure region with customer resources, including a virtual network gateway.

Komprise stands out for its dynamic data analysis, transparent archiving, and direct data migration capabilities that allows uninterrupted access to hot data and user activities. Komprise’s HyperTransfer technology further amplifies these capabilities, ensuring high-speed data transfers which are crucial for large-scale migrations and data management tasks.

Komprise also maintains data availability and reliability during an active migration by working alongside the storage platforms to continually compare source and target storage looking for changes to apply, as well as including a hashing mechanism to validate data has been stored and read back properly as it’s moved.

Management with Komprise is via the Komprise Director, a management console which orchestrates data-movement activities. It provides centralized control, enabling users to manage, analyze, migrate and monitor activities across the customer’s total storage landscape, including single, or multiple sites.

At the heart of the Komprise architecture are the Komprise Observers and Proxies. These are the data movement engines which can be strategically placed to ensure high performance during data management and migration operations. Typically, the Observers are positioned in proximity to legacy storage systems for fast local access benefiting local migrations. For migration to Azure ANQ cluster, Komprise HyperTransfer leverages Proxies hosted within the Azure tenant to handle writing and validating data locally in Azure. This architecture enables efficient data transfer over WAN significantly reducing latency when compared to common migration tools. Note: the data transits only networks that are secured and managed directly by the customer.

The ANQ instance is deployed in Qumulo’s Azure tenant.

Access to the ANQ cluster is enabled via VNet injection from a dedicated subnet in the customer’s Azure tenant, which connects to the customer’s ANQ service in the Qumulo tenant. All data on ANQ is accessible only via the network interfaces in the customer’s delegated subnet. Note: Qumulo has no access to any data on any ANQ service instance.

The HyperTransfer technology integrated into the Komprise suite is pivotal in accelerating the data movement process. It optimizes network usage to transfer large datasets rapidly, significantly reducing the time required for data migrations and synchronizations. This efficiency is critical for organizations looking to minimize downtime and quickly leverage cloud scalability and flexibility.

Solution Workflow

A migration project consists of 5 steps:

Step 1: Analyzing Your Data

Komprise provides quick insights into your unstructured data across all storage platforms with Plan Analysis and Deep Analytics capabilities. Plan Analysis immediately gives summary results with Usage Graphs and the Analysis Activities pages which surfaces important file system issues discovered customers will want to know before moving data. Deep Analytics allows customers to dig deeper into understanding their data with custom querying capabilities and graphs to find select data sets, orphaned files and more.

The Komprise tool’s Analysis Activity page helps identify potential issues upfront, before moving data. Potential issues to watch for before initiating the migration sequence might include:

Knowing Files and Directories stats, and identify any restricted access or resolution issues
Are there exceedingly long path and file names, do they contain invalid characters
Identify exceedingly large numbers of tiny files, if there are large numbers of empty directories
Identify potentially slow-performing shares
Know up front if there are sparse files, or cyclic symbolic links cross connecting directories

Step 2: Assessing Network and Storage Performance

Migrations move only as fast as the infrastructure allows. It’s vital to know the combined performance abilities of the network and storage systems together. Measuring network and storage performance separately may not reveal hidden limitations in port configurations, routing, file system overloading and more.

Migrations can place an atypical and very heavy workload on the customer’s network. Komprise knows the importance of assessing combined network and storage performance to identify any connectivity issues between the legacy NAS system and the ANQ instance before moving data.

The Komprise Assessment of Customer Environment (ACE) tool is easy to deploy and run. The tool simulates a series of data movement scenarios between on-premises source legacy NAS shares and the destination ANQ instance. It performs a set of reading, writing and checksum operations, and collects the resulting overall performance numbers. These results can highlight potential performance risks and infrastructure bottlenecks to investigate to ensure the network topology is ready.

Step 3: Migration: Initial Synchronization

The migration process kicks off with Komprise capturing an extensive analysis of the overall dataset to be migrated, and performs the initial movement of data from the legacy NAS system to the ANQ target. This first iteration establishes a foundational baseline and is pivotal for the initial, substantial data transfer, which occurs without disrupting users’ access to the source data. Komprise leverages HyperTransfer technology to ensure an optimized transfer process by dynamically adjusting the parallelization based on Observer and Proxy resources available and available network bandwidth.

Step 4: Migration: Incremental Iterations

In this phase, Komprise’s intelligent algorithms come into play to track and transfer only those changes made to the source data since the last synchronization. This approach is vital for keeping the target storage updated with minimal network disruption. It reflects an enhanced understanding of data activity and ensures a more precise and efficient synchronization process. View this iteration as a trial run for a final cutover. You know where you stand with full system file comparison check showing any errors, total transfers times, amount of data moved and more. This information helps to plan when to schedule a final cutover and duration.

Step 5: Migration: Final Iteration (Cutover)

This crucial phase marks the completion of the migration, with Komprise transferring any remaining data to the target storage. To ensure an exact replication of the source at the point of cutover, write access to the source NAS is temporarily suspended for a last file system check. This step underscores the importance of thorough planning and coordination to minimize impact on operations.

Komprise Capabilities

Below are some items that Komprise includes with all migrations.

The Komprise Cloud Director oversees the deployment and management of Komprise data management processes. This single management interface helps customers manage multiple migration operations across the whole organization. From a single console you can initiate migrations, schedule migrations for later start time, see a clear status tracking showing the schedule and movement of data between legacy NAS systems and ANQ, track progress, collect progress reports, review issues and plan cutovers.
Komprise intelligently pools resources to distribute the overall workload and provide redundancy, enhancing throughput and efficiency, which is critical for shortening the time required to migrate large datasets. Komprise can dynamically scale up existing resources, as well as scale out, adding additional Observers and Proxies.
Supports secure data transfers with in-flight encryption and using the customer’s connectivity between on-premises environments and Azure-based resources facilitated through Azure VPN Gateway or ExpressRoute. This combined with Komprise’s HyperTransfer technology ensures secure accelerated transfers and efficient data movement.
Komprise should run using an Active Directory user on both the legacy NAS system and the ANQ instance to ensure full access to all files for SMB shares and to ensure the transfer of NTFS and ACL permissions.
If the source NAS system includes both SMB and NFS data, it is possible to migrate datasets that include NTFS and POSIX permissions. When handling legacy NAS systems containing both SMB and NFS data, Komprise can manage migrations while preserving NTFS and POSIX permissions.
Cross-protocol permissions, in which a single dataset is shared via both POSIX and ACL permissions, may require an additional step to migrate successfully. For the initial synchronization, the NFS-first approach (and migration of the data’s POSIX permissions structure) is usually the quicker method due to the lower overhead of the NFS protocol. Subsequently, an additional migration task can be created to apply the NTFS permissions to the same data.
Komprise simplifies the selection of source directories from legacy NAS systems for migration to ANQ, intelligently excluding system-specific directories such as .snapshot or .snapshots to ensure a clean transfer.
Komprise employs parallelization to expedite the migration process, creating multiple concurrent data streams. This process is optimized by HyperTransfer technology, which dynamically adjusts to maximize the use of available bandwidth and minimize migration windows.
Komprise ensures data integrity throughout the migration process, utilizing appropriate hashing algorithms to verify data without sacrificing performance.

Components

The solution architecture comprises the following components:

Azure Native Qumulo Scalable File Service cluster to provide consolidated data storage for all content created using the solution

Komprise Cloud Director to migrate data from local storage to ANQ.
- Observers
- Proxy Servers – consisting of Komprise Proxy Services running on Windows Server with (minimum 16 core CPU, 32 GB memory, and 10 GbE network)
Azure ExpressRoute and/or Azure VPN Gateway
Azure VNet Injection
Azure Virtual Network
Legacy NAS (e.g. on-premises Dell EMC Isilon / PowerScale)

Considerations

File migration to Azure Native Qumulo Scalable File Service with Komprise should include the following considerations in their planning and design processes.

Potential Use Cases

Data Migration: Cloud migration is the process of moving applications and data from on-premises systems to a public cloud. However, legacy NAS options in the market have limited functionality or scalability on the public cloud. ANQ offers enhanced functionality and scalability on Azure. Komprise facilitates the high-performance migration of your file data to ANQ while ensuring data integrity.
Disaster Recovery: ANQ can be utilized as a secondary DR site, allowing organizations to replicate their data to the cloud. In the event of a disaster that impacts the primary on-premises site, organizations can switch to the replicated data on ANQ and continue operations with minimal disruption. Komprise quickly detects files to be replicated from a source legacy NAS.
Tiering: Komprise’s tiering capabilities offer a seamless solution for managing and optimizing file storage by intelligently analyzing and moving inactive or cold data to ANQ, leveraging Komprise Transparent Move Technology (TMT). This technology ensures that data is moved without disrupting user access, maintaining native file access and enabling direct access in ANQ. The unique value lies in its ability to reduce storage and backup costs, simplify data management with intuitive dashboards and API integration, and provide a flexible, no lock-in path to the cloud, making it highly useful for customers aiming to efficiently manage data growth and cloud storage costs.

Scalability and Performance

When planning a data migration solution using Qumulo and Komprise, enterprise architects and other stakeholders may want to include the following in the solution:

Capacity and growth – ANQ scales easily on demand, allowing you to add as much capacity as needed simply by creating or migrating data.
Komprise allows you to migrate files, folders, and ACL data from legacy NAS to ANQ with full automation, load balancing, and macro and micro-level reporting. We can scale the power of our tools thanks to multi-streaming, multi-threading Komprise Observers and Proxy Servers.
Throughput – ANQ allows you to adjust throughput on demand in 1GB/s increments to ensure the availability of the throughput you need at all times.
With the Komprise, the transfer rates between the source and target storage can approach the end-to-end connection’s network speed. Performance can be impacted by data type and layout: a large number of small files or a high directory count within a given dataset can slow down overall throughput by increasing the load on the Proxy machines. Also, the use of Komprise’s HyperTransfer technology to its fullest extent for accelerating data transfers.

Security

The Azure Native Qumulo Scalable File Service connects to your Azure environment using VNet Injection, which is fully routable, secure, and visible only to your resources. No IP space coordination between your environment and the ANQ cluster is required.

Care should be taken during design and implementation to ensure that the solution’s security complies with industry best practices, internal enterprise policies, and any applicable legal/regulatory requirements.

For all other aspects of the solution, customers are responsible for planning, implementing, and maintaining the security of the solution to satisfy all applicable legal and regulatory requirements for their industry and location.

Cost Optimization

Cost optimization refers to minimizing unnecessary expenses while maximizing the value of the actual costs incurred by the solution. For more information, visit the Overview of the Cost Optimization Pillar page.

Azure’s pay-as-you-go model allows you to optimize costs by scaling resources to use the capacity when needed. This helps you manage costs efficiently, without over-provisioning resources.
The cost of the Qumulo depends on the amount of data on the Azure Native Qumulo Scalable File Service and the performance tier. For details, see Azure Native Qumulo Scalable File Services pricing.
Komprise’s architecture is designed for scalability and to provide parallelized data flows. When you need more power and higher performance, you just need to add another Observer or Proxy to match your needs and consider only that VMs resources costs.

Availability

Different organizations can have different availability and recoverability requirements even for the same application. The term availability refers to the solution’s ability to continuously deliver the service at the level of performance for which it was built.

Data and storage availability

The ANQ deployment includes built-in redundancy at the data level to ensure data availability against failure of the underlying hardware. To protect the data against accidental deletion, corruption, malware, or other cyberattack, ANQ includes the ability to take snapshots at any level within the file system to create point-in-time, read-only copies of your data.

ANQ supports replication of the data to a secondary Qumulo storage instance, which can be hosted in Azure, in another cloud, or on-premises. ANQ is compatible with file-based backup solutions to enable external data protection.

Service stability and continuity

Depending on the scope and criticality of the data being migrated, a phased approach may be appropriate in some circumstances. Rather than migrate the entire contents of your legacy storage in a single operation, a phased approach would entail migrating one workflow at a time, using smaller, less-critical datasets to develop and finalize the process before moving to larger and more-important data.

An added benefit of this approach is that any issues that may arise from a migration operation are limited in scope to a single workflow or dataset.