Unlock the power of your enterprise data with Amazon Q Business

NetApp is announcing the general availability of the NetApp® Connector for Amazon Q Business. This innovative solution seamlessly connects enterprise data from enterprise file systems with Amazon Q Business, helping employees to access relevant information quickly, generate context-rich content, and boost productivity with data-driven insights. The Connector is deployed and managed with NetApp® BlueXP™ workload factory for GenAI service, giving our customers a unified single experience to extend enterprise data for generative AI use cases. Developers already use Knowledge Bases to process and integrate enterprise data with applications by using retrieval-augmented generation (RAG) framework. And now, with the NetApp Connector for Amazon Q, data owners can easily connect enterprise data to Amazon Q Business applications.  

Amazon Q Business leverages generative AI to provide rapid insights and actions for knowledge workers by connecting to multiple enterprise data sources. Enterprises often store vast amounts of unstructured data on corporate file shares. However, integrating this data with Amazon Q has been challenging due to the limited availability of file system connectors. Also, connecting unstructured datasets residing on premises and in distributed locations to Amazon Q can add complexity and cost. Traditional methods of connecting data from files on premises often require complex data migrations or the creation of data copies, which can also add unnecessary cost and complexity. And making sure that applications built using Amazon Q honor the authentication and authorization mechanisms and safeguard data privacy can require complex design and architectural setups.

The NetApp Connector for Amazon Q addresses these challenges by providing an easy-to-use, secure, and cost-effective solution to connect enterprise data on NetApp file storage to Amazon Q Business. Customers can get insights from their unstructured file data faster while containing costs and extending their access controls for a secure solution.

A simple yet flexible solution for integrating file systems. Customers can connect enterprise SMB or NFS file shares on NetApp storage and index their data with Amazon Q Business without the need to develop any custom data indexing or data transfer mechanisms. The indexed data is then available to the configured Amazon Q Business application to generate insights that the business users need. Customers can also connect to any network-accessible SMB or NFS share that is not on NetApp storage, and they can index data using the same Connector, providing a flexible solution for heterogenous file environments. Different applications may also make it necessary to select only a few types of files (such as index only Word and PDF documents) and leave the rest. The file filters can be set to index only a subset of document types, providing flexibility to control what data is indexed while reducing the cost of indexing data that is not relevant.

Enhanced security and data privacy. Security is paramount for any enterprise, and the NetApp Connector for Amazon Q Business extends NetApp’s security-first approach. It maintains granular user access permissions set on the source documents, so that users of Amazon Q get insights only from data that they are authorized to see. Customers are also concerned about safeguarding personally identifiable information (PII) and not exposing it to applications. Using the Connector’s seamless integration with NetApp BlueXP™ classification, customers can easily configure the data indexing process to scan the contents of the file and redact any PII identified before the files are made available to Amazon Q. Customers can therefore use BlueXP classification to standardize their data discovery for compliance and privacy while extending the solution to Amazon Q Business applications.

Accelerate time to value while avoiding data silos. To connect on-premises data to Amazon Q, customers can use NetApp SnapMirror® and FlexCache® technologies to bring data closer to Amazon Q. Using network efficiency data replication and data caching technologies reduces latency to index the data and also avoids the need to migrate on-premises data or create separate data copies. Many customers use SnapMirror data replication technology to create backup and disaster recovery environments in Amazon Web Services, using Amazon FSx for NetApp ONTAP as the destination for their on-premises NetApp environments. Typically, these backup or DR copies lie dormant. However, they can now be connected to Amazon Q to deliver business insights without creating any additional data management complexity.

Reduce complexity and cost of data refresh. Customers want to make sure that Amazon Q for Business applications always have access to the latest data and pick up any changes to either the connected data sources or the source access controls. Customers also want to lower the cost and time to index changed datasets by avoiding the need to index the whole dataset every time the data changes. NetApp Connector for Amazon Q Business simplifies data refresh by automatically synchronizing source datasets to Amazon Q every 24 hours, or at any time on demand. The first time that data is indexed to Amazon Q, the whole dataset is scanned and indexed. However, the Connector also maintains the metadata of the indexed files, so that for the next synchronization operation only the changed files are reindexed, reducing the overall time and cost to index changes. Additionally, the metadata index is automatically protected by using Amazon FSx for NetApp ONTAP Snapshot™ copies and can be used for rapid recovery from data loss or corruption. After recovery from a snapshot, only incremental changes will need to be index, hence improving recovery time and reduces costs.

The NetApp Connector for Amazon Q is deployed and managed by the NetApp BlueXP workload factory for GenAI service. The Connector extends the current workload factory GenAI capability that enables customers to create and manage knowledge bases for advanced data processing and indexing by using the RAG framework (Figure 1).

Figure 2 shows the overall reference architecture for how the Connector indexes the data as well as the user query flow.

Connector setup and indexing workflow.

(1) To set up the Connector, customers first need to have Amazon Q and an Amazon Q application configured in their AWS account. When the Amazon Q application is configured, customers can use the BlueXP workload factory interface to deploy the NetApp AI engine infrastructure within their AWS account and VPC. The Connector can also use an existing AI engine used for knowledge base infrastructure.

(2) (Optional) If the data that needs to be connected to Q resides on premises, NetApp SnapMirror or FlexCache technology can be used to replicate or cache data on an FSx for ONTAP file system.

(3) The NetApp connector then scans the connected data sources and indexes the files to an Amazon Q index connected to the Amazon Q application. Supported data sources include FSx for NetApp ONTAP volumes; data protection for FlexCache volumes created with SnapMirror or FlexCache relationships from on-premises NetApp systems; and any SMB shares or NFS exports that are accessible from the customer’s VPC. Customers can also set up file filters to select only certain file types to be indexed into Amazon Q.

(4) When the data sources are configured, the Connector securely mounts the configured SMB shares or NFS exports, creates an index of the source files, and stores the index metadata on a persistent volume provided by an FSx for ONTAP file system in the customer’s VPC.

(5) For connected SMB shares, the Connector also gets emails of users who are part of the file access control list. Amazon Q uses the user’s email to enforce access controls on the files indexed by the NetApp Connector.

(6) (Optional) Customers who have BlueXP classification deployed on their VPC can configure data guardrails to find and redact any PII from source files. In this case, the Connector reads the file contents and sends them to BlueXP classification for analysis. If any PII is found, all occurrences are replaced by <PII_REDACTED>> and the cleaned files are sent to Amazon Q index. The Connector does not make any changes to the source files.

(7) Finally, the files scanned by the Connector are sent to Amazon Q for indexing and search by the Amazon Q applications.

User query workflow.  

(1) A user accessing Amazon Q web experience is authenticated by the IAM Identity center.  

(2) The authenticated user sends a query to Amazon Q.  

(3) Amazon Q index performs a search of the relevant documents and uses the user’s email to enforce searching only across documents that the user has access to.   

(4) When the relevant documents are identified, the information from these documents is used to generate a response back to the user. 

The NetApp Connector for Amazon Q is a game changer for organizations that are looking to harness the power of their enterprise data. By simplifying data integration, enhancing security, and leveraging generative AI, the Connector helps employees to make data-driven decisions, generate new content, and improve productivity. 

For more detailed information, including how to set up and use the Connector, read the product documentation. To get started today, create an account and log into the NetApp BlueXP workload factory console.  

The blog is co-written by Puneet Dhawan and Yuval Kalderon.