NetApp XCP Data Mover Solution: On Premises to Cloud
Karthikeyan Nagalingam
To transform their enterprises, many businesses are moving from on-premises infrastructure and applications to cloud-based storage and applications. Such a move can help them improve their data management, enable application mobility, and improve data governance.
NetApp XCP is client-based software that uses all the available system resources to handle high-volume datasets and high-performance migrations.
This blog describes two data migration scenarios that leverages NetApp® XCP as the data mover/migration solution. Both use cases involve moving data from an on-premises environment to the cloud. In one use case, the customer needed flexible cloud storage with high performance on demand; in the other use case, the customer needed to migrate large files to Azure. For both customers, time was critical.
Use Case 1: Using the XCP Data Mover to Migrate Millions of Small Files to Flexible Storage
This use case is based on our largest tourism industry customer for on-premises-to-cloud data migration. Because COVID-19 has caused demand in the travel industry to decrease, the customer wanted to save capital expenses on high-end storage in their on-premises environment for its demand pricing application. The customer had a tight SLA to migrate millions of small files to the cloud.The customer had the following challenges and requirements:
- They needed to migrate about three hundred and fifty million small files from the on-premises environment to Azure.
- The customer needed flexible storage in the cloud, with high performance that would match that of the on-premises environment.
- The customer wanted to do a one-time migration.
- The customer has a predictable pattern of high demand for several months of each year, which slows down for just a few months of each year. They needed flexible storage so that they could choose the right storage service level according to performance demand.
In our solution, the customer had to move several million files to Azure over the weekend by using Azure ExpressRoute. XCP transferred the data with parallel processes. First, we built the XCP setup with a single instance and transferred a couple million files to Azure NetApp Files. This approach helped the partner (Microsoft) and the customer determine how much time would be needed to copy the complete set of files. For the same test, we chose the basic service level for Azure NetApp Files, which provides 16MBps/TB throughput. The premium service level provides 64 MB/s/TB. The Ultra service level provides 128 MB/s/TB. We changed the XCP Azure instance networking to accelerated networking enabled, and we changed the capacity pool and service level to match the throughput requirement, because the performance (throughput) scales linearly as volume size grows by capacity. Then we worked with the customer to decide the number of subfolders per XCP instance. We chose the number of XCP instances to finish the migration of three hundred and fifty million files over the weekend. The size of the data set is around two and half terabytes. To make the data transfer faster, we chose the ultra-service level Azure Netapp Files volumes for some XCP instances destination location.
The data migration process provided the following benefits:
- The customer was able to migrate data from its on-premises environment into Azure within the time required.
- Multiple XCP instances helped the customer get maximum performance by aggregating the throughput of multiple network interfaces and server resources from multiple data sources.
- The XCP license is free, so there was no cost for the data migration.
- NetApp worked with the customer to create a migration script for the data migration from the folders list so that no manual intervention would be needed.
- Using an XCP Linux virtual machine instance, the customer was able to transfer millions of NFS files rapidly.
Use Case 2: Using the XCP Data Mover to Migrate Large Files
This use case is based on a television network customer. The customer wanted to migrate Oracle Recovery Manager (RMAN) backup files to the cloud and run Oracle E-Business Suite (EBS) application by using Azure NetApp Files with Pacemaker software. The customer wanted to migrate their database backup files to the cloud on demand. Customer wanted to transfer large files (file size is in the range of 25GB to 50GB each) to Azure.
The customer had the following challenges and requirements:
- Based on demand, the customer wanted to copy the Oracle Database backup files, which were in TAR format. The files were large and needed to be in Azure in a short time.
- The customer needed better data transfer performance than the operating system copy offered, so that they could provide the Oracle backup data to the EBS application at the right time. Time was key for this customer.
The data migration process provided the following benefits:
- The customer reported that XCP is simple to configure and easy to use.
- The transfer of Oracle Database backup files was faster than with the operating system copy for Oracle RMAN backup files.
- The customer reported that their previous on-premises-to-cloud copying was always a bottleneck, and they had needed extra time and costly products for the migration. In contrast, XCP performance removed those roadblocks and streamlined the data migration.
Learn More
To learn more about the NetApp XCP please visit our page.
Karthikeyan Nagalingam
Karthikeyan Nagalingam is a Principal Technical Marketing Engineer at NetApp for NetApp XCP, Fpolicy, Filesystem Analytics and Antivirus. His previous roles in Emerging Technology Solutions involved in Pre-Sales and Post-Sales technical activities with fields, partners and customers. He holds an Master of Science in Software Systems from Birla Institute of Technology and Science.
