NetApp StorageGRID to modernize data management

Modern IT infrastructure must be flexible, scalable, and resilient—especially when managing massive volumes of data. At NetApp IT, we’ve taken on this challenge head-on by evolving our use of StorageGRID to meet the increasing needs of our organization and our customers. From supporting global file uploads to transforming our data lake architecture for Active IQ, StorageGRID has become a cornerstone of our data infrastructure strategy.

When we first introduced StorageGRID over a decade ago, it was a cautious step. We started by backing up static ONTAP data using NetBackup and AltaVault. Once AltaVault was sunset, we retained the environment to meet SOX retention requirements. StorageGRID’s object storage capabilities soon proved ideal for more than just backups—it became a reliable backend for Splunk logs, Git repositories, and Veeam VM backups.

Today, our architecture includes three distinct StorageGRID environments:

• Corp StorageGRID (Internal-only, 6PB, 67 nodes)
• DMZ StorageGRID (Customer-facing, 6PB, 23 nodes)
• Bare-metal StorageGRID for Active IQ (6PB, 24 nodes)

These environments are deployed across multiple sites, with careful consideration given to security posture, load balancing (via Avi or VM gateway nodes), and hardware lifecycle management. 

A recurring challenge for NetApp Support was enabling customers to securely upload large core files (up to 2TB) for troubleshooting ONTAP issues. The original browser plugin-based solution was clunky, error-prone, and frequently failed, particularly for customers outside the United States.

We designed a new browser-based upload system using StorageGRID and key platform services:

• Multi-part uploads allow pause/resume capabilities and faster transfers.
• Pre-signed URLs grant secure, session-based access.
• CloudMirror replicates uploaded files from DMZ to our internal grid without compromising security.
• SNS (Simple Notification Service) notifies our system immediately when a file is uploaded, triggering an automated workflow to move it into the engineering domain.
• Cloud Sync transfers files from AWS S3 into internal StorageGRID instances when needed.

This architecture improved reliability and enabled us to scale globally, utilizing AWS S3 as a transient upload platform in regions without data centers. Once data lands in AWS, it’s replicated to the U.S. via the AWS backbone and quickly moved into StorageGRID. 

The result? Seamless, secure uploads for customers worldwide, without plugins, firewall exceptions, or infrastructure changes.

Our second primary use case involved migrating Active IQ’s data lake from a massive Hadoop-based architecture to a Kubernetes-native Dreamio platform backed by StorageGRID.

• The numbers tell the story:
• Compute resources dropped by 60%
• Storage footprint reduced by 2PB
• Query times improved 10–20x, with the largest query going from 45 minutes to 2.5 minutes

We deployed a bare-metal StorageGRID environment using E-Series storage arrays reclaimed from the Hadoop system. We optimized for performance, resiliency, and maintainability using erasure coding (4+2) and Red Hat 9.4.

NetApp IT’s journey with StorageGRID mirrors our customers' challenges: legacy infrastructure, security constraints, evolving requirements, and tight budgets. We’ve learned that:

• Platform services matter – tools like CloudMirror and SNS unlock automation and flexibility.
• Hybrid design is key – balancing internal grids with cloud-native services extends reach without excess cost.
• Reuse accelerates ROI – reusing hardware and smartly transitioning workloads keeps innovation cost-effective.

We’re not just building solutions for today but laying the foundation for future scalability and agility.