Co-authors: Naveen Krishnamurthy from VMware, Jayamohan Kallickal from Broadcom
General availability (GA) of VMware vSphere 7.0 is a major milestone for VMware. The new release supports many new features that can facilitate transformation of modern data centers. One key feature available with vSphere 7.0 is NVMe over Fabrics (NVMe-oF) support.
Broadcom, NetApp, and VMware have a long history of developing infrastructure technologies that modernize data centers. NVMe over Fabrics solutions based on Fibre Channel (FC) deliver the next level of performance required by traditional and real-time applications for various use cases such as AI, machine learning, and real-time analytics.
Highly optimized storage solutions have become a key piece in the digital transformation puzzle. In the first phase of storage modernization, administrators embraced flash storage to accelerate business-critical applications. Increasingly, organizations are looking for further ways to improve the speed and responsiveness of the applications that control their key operations. Because the performance of these applications is tightly linked to time to value and to stakeholder satisfaction, it’s critical that they operate at the highest levels of efficiency. Getting value and insights quickly and reliably from a range of mixed-workload environments can differentiate organizations and accelerate efficiency.
Enter NVMe (NVM Express), a new protocol for accessing high-speed storage media that offers many advantages over legacy protocols. NVMe has become the industry-standard interface for PCIe SSDs, with a streamlined protocol and command set and with fewer clock cycles per I/O operation. NVMe supports up to 64,000 queues and up to 64,000 commands per queue, which makes it more efficient than the existing SCSI-based protocols such as SAS and SATA. The NVMe-oF standard was created to extend the benefits of NVMe to shared storage. It was designed to support many transport protocols, including FC, RDMA over Converged Ethernet (RoCE), InfiniBand, and TCP. The NVMe/NVMe-oF protocol provides very high throughput, low CPU utilization, and ultralow latency, all of which required creating a lightweight NVMe stack that is parallel to SCSI, as well as a redesign of drivers. The NVMe-oF architecture supports many hosts connected to multiple targets through multiple switches.
Most enterprises already entrust their mission-critical applications to FC SANs because of their consistent performance, reliability, and resilience. As your organization considers accelerating business-critical applications, VMware, Broadcom, and NetApp have partnered to create a complete server-to-fabric-to-storage solution that is easy to deploy.
NVMe over Fibre Channel (NVMe/FC) brings the following enterprise-class storage services to NVMe-oF:
The new solution comprises VMware vSphere 7.0, Broadcom Gen 6 host bus adapter (HBA) and fabric, and NetApp® flash storage, a complete nondisruptive upgrade from SCSI to NVMe/FC. NVMe/FC not only accelerates business-critical applications; it also enables simplicity of operations with capital expenditure (capex) reduction.
- Centralized discovery
- Sharing of discovery data across the network
- Zoning and isolation
- State-change notifications
- Storage network topology
- Sequence-level error recovery
Most enterprise data centers already deploy FC SANs, so you don’t need a rip-and-replace upgrade to move to an NVMe/FC solution. Broadcom’s Emulex FC HBAs and Brocade switches support NVMe/FC and SCSI Fibre Channel Protocol (FCP) simultaneously on the same port and fabric. This support simplifies the creation of multipath SANs that enterprises require. Further, Emulex Gen 7 FC HBAs are capable of round-trip latencies in the range of 10 microseconds while providing multimillion IOPS, making these HBAs an excellent fit for NVMe-oF solutions.
The Broadcom family of Gen 6/Gen 7 Brocade FC switches and Emulex FC HBAs has demonstrated more than 20 years of field-proven performance and reliability, making FC the trusted technology of choice. One of the biggest challenges of managing a dynamic infrastructure of physical and virtual resources is networkwide visibility. You can’t manage what you can’t measure, and you can’t measure what you can't see. Broadcom addresses this challenge through close integration with NetApp and VMware products. Broadcom’s Brocade and Emulex Gen 6/Gen 7 FC products not only deliver 32GFC performance, but also provide deep visibility into the health and performance of the SAN that can help you quickly respond to dynamic business demands. Taking advantage of the increased efficiency of NVMe, combined with the high performance, low latency, and actionable intelligence of Gen 6/Gen 7 FC, you can accelerate IOPS to deliver the performance, application response time, and scalability needed for next-generation data centers.
The NetApp AFF A800 all-flash storage array is a robust scale-out system that combines low-latency performance with comprehensive data management, built-in efficiencies, integrated data protection, multiprotocol support, and nondisruptive operations. It’s designed for the most demanding workloads that require ultralow latency, and it’s the first flash array on the market to support NVMe SSDs and NVMe-oF. It provides end-to-end NVMe connectivity between storage arrays and host servers for maximum bandwidth, high IOPS, and the lowest possible latency. AFF A800 systems offer support and deep integration for enterprise applications, virtual desktop infrastructure (VDI), and database and server virtualization, supporting Oracle, Microsoft SQL Server, VMware, SAP, MySQL, and more.
NetApp ONTAP®, the industry’s leading enterprise data management software, combines new levels of simplicity, flexibility, and security with powerful data management capabilities, proven storage efficiencies, and leading cloud integration. ONTAP 9.7 includes several enhancements that enable you to do more with less time and effort. ONTAP provides best-in-class integrations with vSphere, simplifying storage and data management for VMware environments. ONTAP already has enterprise-class storage for VMware, with support for both NFS and FCP/iSCSI on the same cluster to host VMware datastores. ONTAP also has robust support for VMware features such as Virtual Volumes (vVols), vStorage APIs - Array Integration (VAAI), inline deduplication/compaction/compression, vSphere integration through the Virtual Storage Console, backing up virtual machines (VMs) with NetApp SnapCenter®, NetApp FlexClone®, and VMware Storage Replication Adapter (SRA) plug-ins.
Figure 1) A common SAN configuration.
To compare the performance benefits of NVMe/FC versus SCSI FCP, we built the following test configuration, as shown in Figure 1:
- Dell PowerEdge R740 servers
- vSphere 7.0
- Red Hat Enterprise Linux 8.0 VMs running database software
- Trusted Platform Module (TPM) test suites
- NetApp AFF A800 cluster
- Emulex LPe35000 HBAs
- Brocade G620 switches
Test Results: Better Performance with NVMe/FC
On our test configuration, we ran performance tests on Microsoft SQL Server and Oracle 19c on Red Hat.
Figure 2 shows the performance results with SQL Server: a 115% improvement for NVMe/FC compared with SCSI FCP. NVMe/FC delivers more transactions on the same hardware platform.
Figure 2) Performance improvement of NVMe/FC with Microsoft SQL Server on Red Hat 8.
Configuration: HammerDB 3.2/SQL Server 2017 on a Red Hat Enterprise Linux 8.0 VM running ESXi 7.0 hypervisor; Emulex LPe35002 HBA connected to two NetApp AFF A800 arrays through a Brocade G620 switch.
As depicted in Figure 3, Oracle online transactional processing (OLTP) TPC-C results show a 111% increase in TPM with NVMe/FC compared to SCSI FCP. Similar improvement is seen in NOPM for NVMe/FC.
Figure 3) Performance results of NVMe/FC with Oracle 19c.
Configuration: HammerDB 3.2/Oracle 19c on a Red Hat Enterprise Linux 8.0 VM running ESXi 7.0 hypervisor; Emulex LPe35002 HBA connected to two NetApp AFF A800 arrays through a Brocade G620 switch.
Delivering Enterprise-Class NVMe-oF with NVMe/FC
The consequence of ever-increasing reliance on data will be a never-ending expansion in the size of the global datasphere. Estimated to be 33ZB in 2018, IDC forecasts that the global datasphere will grow to 175ZB by 2025. As your business contends with the perpetual growth of data, you need to rethink how data is captured, preserved, and processed. Performance, economics, and endurance of data at scale are paramount.
NVMe/FC with VMware, Emulex, Brocade, and NetApp delivers performance that is about two times better with just a software upgrade. The joint industry-leading NVMe-oF solution enables you to deploy an infrastructure that supports all of the following goals:
To learn more, check out the Tolly report released on May 1. And take a look at NVA-1145-Design: Modern SAN Cloud-Connected Flash Solution, a new NetApp Verified Architecture published by VMware, NetApp, and Broadcom. This best-practice reference architecture illustrates an optimally configured VMware vSphere 7.0 installation on NetApp ONTAP 9.7 using Broadcom Emulex 32GB HBAs and Brocade 32GB G620 FC switches for an end-to-end NVMe/FC solution for critical SQL workloads. Also, if you manage, administer, purchase, or use VMware with traditional SAN or NAS storage, read NVA-1136-Design so that you can learn how to build the fastest cloud-ready solution for mission-critical virtualized workloads.
- Adopt modern, future-ready technologies nondisruptively.
- Accelerate your mission-critical applications that create business value.
- Simplify operations cost effectively.