AI pipelines and the evolution of flexible data infrastructure

Generative AI (GenAI) transforms how we create, interact with, and understand content. From producing realistic images to writing human-like text, GenAI models like Llama, GPT, DALL·E, and Stable Diffusion have set the stage for business innovation and transformation.  

Talking with customers weekly, it’s clear that they are struggling to keep pace with the dynamic needs of AI business projects and the unprecedented demands on storage and data infrastructure. Data overload is a fact of life for storage architects, as it’s the fundamental driver for real-time insights in product strategy, operational effectiveness, and business innovation. 

At NetApp, we understand these challenges and are at the forefront of addressing them. Our intelligent data infrastructure is designed with AI-driven transformation in mind. Whether your project is small or large, from developing a customer support chatbot for a local retail store or designing an AI-powered fraud detection system for a global financial institution, we have you covered. 

At the heart of today’s AI innovations lies the AI pipeline. AI pipelines are particularly well-suited for applications that involve clearly defined tasks, including document processing, fraud detection, and customer personalization. Simplistically, AI pipelines can be compared to an automotive manufacturing assembly line, where each stage represents an essential step in building a GenAI-based application. It is a set of iterative steps that begins with data ingestion and pre-processing, then traverses through training and fine-tuning, and concludes with inference and monitoring. To deliver meaningful insights and outcomes, each independent stage in the AI pipeline must be tied together in a coherent whole, just like every step in an assembly line.  

The different stages of the AI pipeline have distinct storage performance demands: 

• Data ingestion and pre-processing requires substantial storage capacity and fast sequential write and read speeds to manage large volumes of data. 

• Model training and fine-tuning call for extreme levels of throughput from the underlying storage. 

• Inference needs high read throughput and low latency real-time data access for accurate predictions and decisions. 

With these stages in mind, we began exploring ways to reimagine storage architecture—developing a flexible approach tailored to meet the diverse requirements of an AI pipeline, while complementing our existing storage portfolio. An intelligent data platform that makes your unstructured data searchable, allowing you to unlock insights for decision-making and improve customer experience with your products and services.

To meet the data requirements of resource-intensive stages in the AI pipeline, we are investing in composable storage infrastructure.  

Composable infrastructure isn’t a new concept. Server manufacturers have used composable infrastructure to inject new life into the blade market with limited success, providing flexible resource utilization by carving out infrastructure components—compute, memory, and storage—into pools to eliminate waste, mimicking cloud-resource consumption.  

More recently, composable infrastructure has seen a resurgence in AI storage platforms, an evolution if you will, separating compute nodes (that host the storage OS) from storage nodes (that host the storage devices) to independently scale out resources over high-speed, low latency networks. This allows for autonomous expansion of compute resources for data management and storage functions, to scale storage performance and capacity asymmetrically, and efficiently manage petabytes of structured and unstructured data.  

Independent scaling makes it possible for us to add more sophisticated data services to the compute nodes. Our storage platform therefore transitions to an intelligent data platform that enriches the underlying data to manage complex AI processes within a single unified solution. 

Moreover, having a modern, composable storage architecture allows for simplified tech refreshes for storage and IT architects. The compute nodes can be refreshed independently of the storage nodes, minimizing business disruptions and maximizing resource utilization. 

This next generation distributed storage architecture will provide you with the same ONTAP technologies that you have come to love and trust for over 30 years. Addressing challenges like data fragmentation, security and ransomware protection, disaster recovery, multi-tenancy, multi-cloud data mobility, and data efficiency. 

Composable storage infrastructure is an ideal solution for storage and IT architects seeking to stay ahead in a rapidly changing AI landscape.

• Unified, scalable storage platform. Unified platforms increase the efficiency of AI pipelines by storing vast amounts of data for pre-processing, training, fine-tuning, inference as well as retrieval-augmented generation (RAG) operations. Having a unified storage operating system (OS) provides a consistent experience across public clouds, hosting services, private data centers, and edge environments. This approach reduces data silos, ensuring seamless access to critical datasets, and enabling AI pipelines to drive innovation without unnecessary barriers. 

• Turnkey RAG retrieval. RAG techniques that operate with a large language model (LLM), small language model (SLM), or industry-specific AI model can address the challenge of providing business and domain-specific context without having to retrain the model. This technique permits the LLM to generate outcomes that are augmented with company and industry data that was not part of its initial training and reduces the chance of experiencing hallucinations.  

But getting your data ready for RAG can be complex. There are multiple steps in the pipeline for building RAG-based applications, including ingestion of the right data, change tracking, ensuring it is high-quality data, removing or redacting any sensitive information, creating vector embeddings, and deploying and managing vector databases to enable semantic searches on the data. Managing vector embeddings alone is a big challenge as the volume of the embeddings can quickly expand to an order of magnitude higher than the original data set. A turn-key solution to transform unstructured data for RAG-retrieval in an automated, efficient and secure fashion is therefore necessary to accelerate your AI projects: RAG retrieval in a box. Providing easy access to and transforming your data for RAG helps you build business-specific AI-based applications to enhance customer experience, improve operational efficiency, and drive business innovation. 

• Energy efficiency. GenAI requires immense computational power, leading to significant energy consumption in training and deploying large-scale models. As adoption grows, optimizing energy efficiency in storage infrastructure will be crucial in balancing innovation with sustainability. To decrease carbon emissions, storage vendors are focused on composable infrastructures with high-density disk enclosures that leverage NVMe capacity flash technology, like QLC flash storage, which can cut down on rack space and energy costs. 

In addition, our public cloud partners that leverage NetApp technology for storage services are investing in renewable energy and new rack cooling designs to reduce power consumption in their global data centers. To align with corporate sustainability goals for your AI projects, you can leverage power-efficient cloud compute instances combined with NetApp storage services on hyperscaler infrastructure. 

Quantum and beyond. As AI technologies evolve, NetApp continues to explore next-generation storage architectures to prepare for future breakthroughs. 

But what lies ahead? Future innovations in GenAI promise to go beyond content generation and venture into dynamic, autonomous decision-making systems, with interconnected AI agents taking center stage.  

Unlike static models that simply respond to inputs, these systems operate autonomously, initiating actions, making decisions, and collaborating with other tools to accomplish complex goals. For example, AI agents could act as virtual health assistants, analyzing patient data, diagnosing conditions, and recommending treatments—all while generating detailed medical reports in real-time.  

As these systems mature, they’ll unlock new opportunities, reshape industries, and redefine how we interact with technology. 

It’s no secret that AI workloads are pushing storage architectures to their limits, but with NetApp Intelligent Data Infrastructure, you’re equipped to meet these challenges head-on. By delivering greater performance, scalability, and flexibility, NetApp empowers you to build robust, future-ready AI pipelines and ensures that your storage architecture evolves alongside this dynamic landscape. 

Ready to take your AI storage strategy to the next level? 

Hear more about next-generation data management for AI. 

If you missed out on our webinar where we talked through the survey results of IDC’s AI maturity model white paper, you can watch it on demand. 

To explore further, visit the NetApp AI solutions page.