Empowering the manufacturing and logistics industry with a data-driven platform

In the evolving landscape of logistics and manufacturing, digital transformation is key to staying competitive. This technical blog post explores a hybrid cloud solution on AWS developed through the collaboration of SVA. The solution addresses industry-specific challenges for predictive maintenance tasks.

Integral to this solution is cloud bursting—a feature that’s crucial for manufacturing companies facing variable workloads, especially during peak demand periods. Cloud bursting allows these companies to seamlessly scale their computational and storage resources by extending their on-premises infrastructure to the cloud. This elasticity ensures that manufacturing operations remain uninterrupted and efficient, even under fluctuating demand. By using this hybrid cloud approach, businesses can maintain their core operations on premises—which is stable and predictable—while leveraging the cloud for additional resources as needed. This approach not only optimizes costs by avoiding overprovisioning but also keeps operations agile and scalable, which is vital in the modern manufacturing sector.

Establishing the hybrid connection

The first step involves setting up a connection between the on-premises environment and the AWS Cloud. The configuration must permit communication between both environments.

SVM peering and the FlexCache volume

Next, you need to establish SVM (storage virtual machine) peering between the two NetApp^® ONTAP^® systems, as detailed in the NetApp ONTAP documentation. Then you need to create a NetApp FlexCache^® volume, a task explained in the documentation. To carry out these steps, you’ll need access to the Amazon FSx for NetApp ONTAP command-line interface (CLI).

Mounting the volume to the GPU instance

After these configurations, you can mount the volume with the on-premises data to the GPU-accelerated instance in AWS. This step is necessary for applying GPU capabilities to machine learning tasks.

The NetApp DataOps Toolkit

To integrate NetApp DataOps Toolkit features, consult the guide for installing and configuring the NetApp DataOps Toolkit for traditional/Kubernetes setups, which is available on GitHub. If you aren’t a storage expert, this toolkit is essential for using the data functionalities of FSx for ONTAP efficiently.

Working environments

After the cloud architect configures the environment, data scientists have several options for their working environment, including Jupyter Notebooks on Elastic Cloud Compute (EC2) instances, AWS SageMaker notebooks, or machine learning operations (MLOps) platforms. 

Use case: Jet engines dataset

In this scenario, data scientists work on a predictive maintenance task that uses jet engine telemetry data collected on an on-premises system. The goal is to develop a machine learning model that predicts the remaining useful life (RUL) of these engines (until they need an overhaul) and take advantage of fast GPU access.

Leveraging the NetApp DataOps Toolkit

The DataOps Toolkit is used to import and manage datasets efficiently:

• NetApp FlexClone^® volumes for dataset versioning. Create writable NetApp Snapshot^™ copies that save incremental changes.
• Snapshot technology for safety copies. Use Snapshot copies to maintain dataset integrity.
• Intelligent infrastructure for seamless data exchange. This feature makes data exchange between on-premises and cloud environments.
• FSx for ONTAP storage efficiencies. Work in the background to compress and deduplicate datasets, reducing physical storage requirements.

Data access and management

The solution offers several commands for managing and accessing data:

• Listing volumes/directories. Use list_volumes functionality to visualize team volumes and subvolumes.

• Prepopulating FSx for ONTAP volume. The prepopulate_flex_cache command can be used for faster data access.

• Creating FlexClone volumes. Implement the clone_volume command for efficient data copying.

Data preparation and sharing

To prepare and share data, you can use these features:

• Snapshot creation. Regular Snapshot copies of volumes maintain data integrity and make it easier to collaborate.

• Volume restoration. If data is deleted, the DataOps Toolkit can quickly restore data through NetApp Snapshot technology, minimizing downtime and risk of data loss.

The joint solution from SVA & NetApp harnesses the strengths of both on-premises infrastructure and public cloud platforms, bridging a crucial gap. On-premises setups offer low latency and robust data governance, whereas public cloud platforms provide scalability and advanced capabilities. This hybrid cloud approach allows you to manage resources seamlessly, so manufacturing companies can use machine learning models, process large datasets in real time, and deploy applications with agility.

SVA and NetApp technologies create value by:

• Utilizing infrastructure resources more efficiently across on-premises and cloud environments
• Enabling a simple and automated workflow while integrating smoothly into existing IT operations
• Speeding up data access and improving overall infrastructure performance for data scientists and engineers

For more information, visit Empowering the logistics industry with Data-Driven Platforms.

Thank you to everyone who contributed to this blog. A special acknowledgment goes to our partner, SVA, for their invaluable contributions. Further, special thanks to my colleague, Max Amende, for co-authoring this blog.