NetApp Tech OnTap NetApp Logo
NetApp Tech OnTap
NetApp and Red Hat Collaborate for pNFS
Pranoop Erasani
Technical Director, NFS
Justin Parisi
Technical Marketing Engineer

Access to shared data is critical to the performance of a variety of scientific, engineering, business, and financial applications. NFS—the most widely used standard for shared data access—can become a bottleneck for large-scale compute clusters, which can overwhelm file servers that are the single point of access for all files in a shared file system.

Most of the solutions available to provide high-performance and shared data access have been more or less proprietary, and they have failed to gain the kind of heterogeneous system support and widespread adoption that standard protocols such as NFS have achieved.

The parallel NFS (pNFS) standard—a subfeature of the NFS version 4.1 protocol specification (RFC 5661)—addresses the single server bottleneck and has great promise to become a standardized solution for parallel data access. In this article we explain how pNFS works, talk about efforts that NetApp and Red Hat are making to move pNFS forward, and describe how pNFS is implemented in clustered Data ONTAP.

What Is pNFS?

The pNFS protocol gives clients direct access to files striped across two or more data servers. By accessing several data servers in parallel, clients achieve significant I/O acceleration. The pNFS protocol has been designed to deliver graceful performance scaling on both a per-client and per-file basis, without sacrificing backward compatibility with the standard NFS protocol. Clients without the pNFS extension are still able to access data.

pNFS Architecture and Core Protocols

The pNFS architecture consists of three main components.

  • The metadata server (MDS), which handles all nondata traffic. The metadata server is responsible for maintaining metadata that describes where and how each file is stored.
  • Data servers that store file data and respond directly to client read and write requests. File data can be striped across a number of data servers.
  • One or more clients that are able to access data servers directly based on information in the metadata received from the metadata server.

There are three types of protocols used between the clients, metadata server, and data servers.

  • A control protocol is used between the metadata server and data servers to provide synchronization. This is not defined by the pNFS specification and will vary from vendor to vendor.
  • The pNFS protocol is used between clients and the metadata server. This is essentially NFSv4 with a few pNFS-specific extensions. It is used to retrieve and manipulate layouts that contain the metadata that describes the location and storage access protocol required to access files stored on several data servers.
  • A set of storage access protocols used by clients directly access data servers. The pNFS standard currently defines three categories of storage protocols: file-based (RFC5661), block-based (RFC5663), and object-based (RFC5664). The clustered Data ONTAP® architecture currently supports the file-based storage protocol and uses NFSv4.1 to access the data servers.

Elements of pNFS. Clients request layout from metadata server (pNFS protocol) and then access data servers directly (storage access protocol).

Figure 1) Elements of pNFS. Clients request layout from metadata server (pNFS protocol) and then access data servers directly (storage access protocol).

To access a file, a client contacts the metadata server to open the file and request the file’s layout. Once the client receives the file layout, it uses that information to perform I/O directly to and from the data servers in parallel, using the appropriate storage access protocol without further involving the metadata server. pNFS clients cache the layout until they are done with the parallel I/O operations. pNFS servers have the right to revoke the layout of the file if the server cannot promise parallel access to the servers. Further, pNFS does not modify the current mechanism available in the NFS server for metadata access.

Red Hat and Linux Team for pNFS

A pNFS solution needs both client and server components to function. NetApp and Red Hat have been working together on industry-standard pNFS to develop solutions that solve the problems associated with shared storage.

NetApp addresses the challenges of scale by combining storage clustering and pNFS. NetApp® FAS and V-Series storage running clustered Data ONTAP 8.1 or later can scale from just a few terabytes of data to over 69 petabytes of data, all of which can be managed as a single storage entity. This simplifies management of a pNFS environment and helps eliminate both planned and unplanned downtime.

Red Hat has delivered the first pNFS client that takes advantage of the scalability, flexibility, simplified management, and optimized data paths of pNFS. By providing pNFS in Red Hat Enterprise Linux®, Red Hat enables application workloads to take full advantage of pNFS without modification, allowing a seamless transition for existing applications.

The combination of NetApp storage and Red Hat Enterprise Linux is a first-to-market pNFS solution.

pNFS and Clustered Data ONTAP

NetApp implemented pNFS starting in clustered Data ONTAP 8.1. (There is no 7-Mode or Data ONTAP 7G implementation.) pNFS implemented in clustered Data ONTAP offers a number of advantages.

  • Simplified infrastructure. The overall infrastructure for pNFS is simpler in comparison to other parallel file systems such as Lustre and GPFS, which require many dedicated servers in addition to storage.
  • Manageability. Typically, pNFS includes several file servers that have to be managed separately. Clustered Data ONTAP lets you manage all the server-side pNFS components as a single system.
  • Nondisruptive operations. A pNFS installation on a NetApp cluster benefits from nondisruptive operations for maintenance and load balancing—such as storage failover, LIF migrations, and nondisruptive volume moves—just like any other workload.
  • All nodes can act as metadata servers. In the clustered Data ONTAP implementation, every node in the storage cluster can act as both a metadata server and a data server. This eliminates the potential bottleneck created by having a single metadata server and helps in distributing metadata operations across the cluster.

pNFS on Data ONTAP versus NFS. Every node can serve as both a metadata server and a data server.

Figure 2) pNFS on Data ONTAP versus NFS. Every node can serve as both a metadata server and a data server.

To understand how pNFS works with clustered Data ONTAP, suppose a client mounted a pNFS file system from one node in a cluster. To access a file, it sends a metadata request to that node. The pNFS implementation gathers and returns information that includes location, layout of the file, and the network information needed to reach the location. The client uses the information to access the data directly from the node or nodes where it resides. By providing a direct path to the volume, pNFS helps applications achieve higher throughput and lower latency.

pNFS seamlessly integrates with clustered Data ONTAP nondisruptive operations such as LIF migrate, storage failover, and volume move. If one of these operations occurs, the pNFS client and server automatically negotiate the new direct I/O path to the server, which helps keep throughput the same, all without any disruption to the application. This is a huge benefit for storage administrators because they do not have to explicitly provision network paths to file systems when they carry out maintenance operations on the cluster. Thus, pNFS with clustered Data ONTAP not only helps with performance, but also simplifies administrative workflows during maintenance operations. As you provision and deploy large-sized clusters, this becomes a necessity.

Without pNFS, both metadata and data paths are more or less static. With pNFS, metadata service is distributed across numerous nodes while data paths are direct to the network interface of the node storing the file. When data moves, data paths adapt automatically to maintain optimal performance.

Figure 3) Without pNFS, both metadata and data paths are more or less static. With pNFS, metadata service is distributed across numerous nodes while data paths are direct to the network interface of the node storing the file. When data moves, data paths adapt automatically to maintain optimal performance.

Best Practices

Attention to a few best practices will help deliver the best pNFS performance.

  • Each cluster node supporting pNFS should be configured with at least one logical interface (LIF) so that pNFS clients can access the volumes stored on that node directly.
  • For metadata-intensive workloads, pNFS clients should be configured so that mounts are distributed across all the nodes in the cluster so that they can all act as metadata servers. This can be accomplished using an external round-robin domain name server (DNS) or via on-box DNS load balancing in clustered Data ONTAP.

For more information on deploying pNFS on NetApp storage, refer to TR-4063.

Red Hat pNFS Client

The Red Hat pNFS client was first released in the Red Hat Enterprise Linux (RHEL) version 6.2 kernel in 2011. RHEL6.2 and RHEL6.3 were both labeled as “tech preview” versions of pNFS. RHEL6.4, released in February 2013, is the first general-availability version of pNFS. You can find complete details on using Red Hat clients with NetApp storage running either NFS or pNFS in TR-3183.

pNFS Use Cases

In addition to its obvious applicability for highly parallel science and engineering applications, the unique capabilities of pNFS make it a good fit for a variety of enterprise use cases.

Business-Critical Applications

Business-critical applications by definition require the highest service levels. Storage bandwidth and capacity must grow seamlessly with server requirements. As NetApp storage volumes are transparently migrated to more powerful controllers in the NetApp cluster, the Red Hat Enterprise Linux pNFS client automatically follows the data movement, self-adjusts, and reoptimizes the data path. The net result is near-zero downtime with no server or application reconfiguration required.

Multi-Tenant Storage Solutions

Having parallel data access means that multi-tenant, heterogeneous workloads benefit directly from pNFS. The data resides on the NetApp cluster and is not tied to a specific NetApp controller. With pNFS, the Red Hat Enterprise Linux servers find the optimal data path and automatically adjust for optimum throughput.

Mixed Clients and Workloads

NFSv4.1 and pNFS can provide flexibility for mounting the file system from anywhere in the cluster namespace. Clustered applications can be mounted over pNFS while legacy applications can still be mounted over NFSv3. File systems that are exported from storage can have clients mounted over different flavors of NFS so that they can coexist without making any significant changes to the applications that access the data. This level of flexibility reduces the overhead of frequent change management.

Virtualization Environments

Hypervisors and virtual machines utilizing the Red Hat Enterprise Linux pNFS client are able to maintain numerous connections per session, which spreads the load across several network interfaces. Think of it as multipathing for NFS, without requiring a separate multipath driver or configuration.


NetApp has been a major driver of both NFSv4.1 and pNFS, cochairing the efforts of the working group. In addition, NetApp has authored and edited a significant portion of the NFSv4.1 specification. This is consistent with our commitment to tackle the problems of storage using industry standards.

With the recent general availability of the pNFS client with the release of RHEL 6.4, you can now deploy pNFS for testing and/or production using a combination of Red Hat clients and NetApp clustered Data ONTAP.

 Got opinions about pNFS?

Ask questions, exchange ideas, and share your thoughts online in NetApp Communities.

By Pranoop Erasani, Technical Director, NFS; and Justin Parisi, Technical Marketing Engineer

Pranoop is the lead NFS architect in the NetApp Protocols Technology Engineering organization, where he leads NFS protocol development for Data ONTAP. He was instrumental in architecting pNFS for clustered Data ONTAP. Pranoop is a strong advocate for leveraging NFSv4.1/pNFS in clustered systems. He has participated in numerous discussions related to the pNFS IETF standard and frequently speaks at NFS interoperability events. He acts as a key technical advisor to technical marketing and product management for ongoing customer deployments and storage software solutions.

Justin is based in RTP and spent the past five years in NetApp Global Support as a Technical Support Engineer and Critical Problem Resolution Escalation Engineer. He focuses on clustered Data ONTAP and has developed troubleshooting course material as well as a number of knowledge base articles. His interests cover a wide range of areas, including CIFS, NFS, SNMP, OnCommand® System Manager, Unified Manager, SnapDrive® and SnapManager® technologies, as well as Exchange, SQL Server®, Active Directory®, LDAP, and more, making him essentially a Swiss Army knife of NetApp knowledge.

Tech OnTap
Subscribe Now
Tech OnTap provides monthly IT insights plus exclusive access to real-world best practices, tips and tools, behind the scene engineering interviews, demos, peer reviews, and much, much more.

Visit Tech OnTap in the NetApp Community to subscribe today.

Learn More About pNFS

Would you like to learn more about pNFS and how to deploy it to accelerate your IT environment? Check out these links:

Contact Us   |   How To Buy   |   Feedback   |   Careers  |   Subscriptions   |   Privacy Policy   |   © 2013 NetApp