As we saw in the previous article in this series, with OneFS 9.15, any node in a PowerScale cluster can now act as the OneFS metadata server (MDS), or serve data connections (DS). This separation of the metadata path from the data path allows file I/O traffic to be distributed across multiple nodes in the cluster, thereby maximizing available bandwidth and parallelism.
In this article, we turn our attention to the enablement, configuration, and management of pNFS.
When a pNFS‑capable client mounts an NFS export on a pNFS‑enabled OneFS cluster, each node functions as both a Metadata Server and a Data Server, with the Metadata Server role being implicitly assigned to the node the client initially contacts during the mount.
Data I/O operations are performed using NFSv3, as permitted by the pNFS Flexible File Layout specification, while NFSv4.1 or NFSv4.2 is used exclusively for layout management and metadata operations. Layouts are granted on a per‑file basis, with the client requesting a layout upon file open that specifies the appropriate data server or servers for that file. Different files may be assigned to different nodes, and traffic distribution behavior is influenced by server‑side alignment settings. The set of data server addresses provided to the client is determined by the SmartConnect pool associated with the mount IP address.
In order for pNFS to activated, a cluster must be running OneFS 9.15, with NFSv4.1 or NFSv4.2 enabled for metadata operations and NFSv3 enabled for data operations. The SmartConnect pool must also use static IP allocation.
| Attribute | Details |
| OneFS version | OneFS 9.15 to provide pNFS support. |
| NFS protocol versions | NFSv4.1 or NFSv4.2 must be enabled (for the metadata path). NFSv3 must also be enabled (for the data path). |
| SmartConnect pool type | The SmartConnect network pool hosting pNFS must use Static IP allocation. |
| Client OS | Linux with kernel pNFS Flexible File Layout support (available in mainline Linux kernels 5.14+). |
| Client mount options | The client must mount the cluster export using NFSv4.1 (-o vers=4.1) or NFSv4.2 (-o vers=4.2). Transport protocol may be RDMA (proto=rdma) or TCP (proto=tcp). UDP is unsupported. |
On the client side, systems must be running Linux kernels that support the pNFS Flexible File Layout, which is available in mainline kernels version 5.14 and later. Additionally, mounts must explicitly specify NFSv4.1 or NFSv4.2 using either TCP or RDMA transport, as UDP is not supported.
With OneFS 9.15, pNFS configuration can be performed via the OneFS CLI or platform API (pAPI), but the WebUI does not currently have an equivalent. Under the hood, the configuration is managed by the ‘isi_gconfig’ registry keys under the NFS driver path. Client remounts are strongly recommended when enabling or disabling pNFS, as clients cache file attributes that determine layout eligibility, so may otherwise retain stale information. Activation requires unmounting NFS clients, enabling the pNFS feature from the OneFS CLI, and restarting the NFS service, with a corresponding process to disable the feature.
Several configuration parameters control pNFS behavior, including a master enablement flag for advertising Metadata Server and Data Server capabilities, alignment settings that determine how file data is distributed across nodes, options for mapping network interfaces to pNFS devices, and internal settings governing layout tracking scalability. In particular, the stripe alignment configuration has a substantial impact on I/O distribution patterns and must be carefully considered when tuning performance, as different alignment values favor different workload characteristics.
| Facet | Details |
| Layout | Layouts are per-file. Each time a client opens a file, it requests a layout that tells it which data server to use. Different files may be assigned to different nodes. How the client spreads its traffic also depends on the server settings around alignment. |
| Network | SmartConnect determines the pool. The set of data server addresses returned to the client comes from the SmartConnect pool that the client’s mount IP belongs to. |
| Node roles | Every node in the cluster acts as both a Metadata Server and a Data Server. The ‘MDS’ role simply means the node the client happened to connect to for that mount. |
| Protocol version | Data I/O uses NFSv3. The pNFS Flexible File Layout standard allows data servers to speak NFSv3, which is well-supported and performant. The client uses NFSv4.1 only for metadata and layout management. |
Every node in the cluster functions as both a Metadata Server (MDS) and a Data Server (DS), with the MDS role simply representing the node to which the client initially connects during the mount operation. Data I/O operations are performed using NFSv3, as permitted by the pNFS Flexible File Layout specification, while NFSv4.1 is used exclusively by the client for metadata handling and layout management. Layouts are granted on a per‑file basis, such that each time a file is opened, the client requests a layout identifying the appropriate data server, allowing different files to be serviced by different nodes, with overall traffic distribution further influenced by server‑side alignment configurations. The set of data server addresses provided to the client is determined by SmartConnect, with the addresses drawn from the SmartConnect pool associated with the client’s mount IP.
When it comes to enabling pNFS, ensure that the following prerequisites are met:
| OneFS version | OneFS 9.15 or later. |
| NFS protocol versions | NFSv4.1 or NFSv4.2 must be enabled for the metadata path. Additionally, NFSv3 must also be enabled for the data path. |
| SmartConnect pool type | The pool must use static IP allocation. Dynamic IP pools are not currently supported for pNFS because device addresses must remain stable. |
| Client OS | Linux with kernel pNFS Flexible File Layout support (available in mainline Linux kernels 5.14+). |
| Client mount options | The client must mount using either NFSv4.1 (-o vers=4.1) or NFSv4.2 (-o vers=4.2). Transport protocol may be RDMA (proto=rdma) or TCP (proto=tcp). UDP is unsupported. |
OneFS pNFS support can be enabled via the CLI as follows:
- First, verify that the desired SmartConnect network pool is configured for static IPs:
# isi network pools list
- Next, check that NFSv3 and NFSv4.1 and/or NFSv4.2 are enabled:
# isi nfs settings global view
- When both the above prerequisites are confirmed, activate pNFS as follows:
# isi nfs settings global modify --pnfs-enabled 1
- Once done, the cluster’s NFS exports can then be mounted from a Linux client running kernel 5.14+. For example:
# mount -t nfs4 -o vers=4.1 <pool-ip>:/ifs/data /mnt/pnfs
- From the NFS client, after performing some I/O, the ‘mountstats’ file contents or CLI command can be used to verify pNFS is active. For example:
# grep -A 20 "pnfs" /proc/self/mountstats
Or:
# mountstats --nfs /mnt/pnfs
Specifically, the presence of non-zero ‘pNFS_read’ and ‘pNFS_write’ counters in the output confirms that the Linux client is using pNFS.
Optionally, as a second part to step 3 above, NIC-level device granularity configuration can also be applied. For example:
# isi_gconfig registry.Services.lwio.Parameters.Drivers.nfs.NFSV4LayoutDeviceUseNICAsUniqueDevice=1
Followed by an NFS service restart:
# isi services nfs disable && isi services nfs enable
Additionally, if pNFS does need to be disabled for any reason, first unmount any NFS clients, then run the following CLI command:
# isi nfs settings global modify --pnfs-enabled=true
The OneFS pNFS implementation (and the pNFS spec in general) does not require any changes to applications, as it is negotiated transparently between the NFS client and server. This allows applications to continue reading and writing files normally while the client kernel automatically manages layout negotiation and communication with data servers.
If a data server node becomes unavailable while a client holds a layout, the client may encounter I/O errors on the NFSv3 data path to that node. However, most Linux NFS clients will fall back to performing I/O through the metadata server (MDS) until the data server becomes available again, at which point they can re-request a layout, with the exact behavior varying by client kernel version and implementation.
OneFS pNFS is supported with both NFSv4.1 and NFSv4.2 and can coexist with non-pNFS clients on the same export, since pNFS is negotiated individually during session setup and clients that do not support it will continue using traditional NFS I/O paths while still accessing the same files.
Additionally, pNFS can work with RDMA when the server advertises both TCP and RDMA data server addresses and the environment includes RDMA-capable interfaces, enabling compatible clients to use RDMA for the data path.
Finally, pNFS requires NFSv3 to be enabled because the Flexible File Layout specification allows data servers to be accessed via NFSv3. The data path between clients and data server nodes uses NFSv3 for simplicity and compatibility, while the metadata operations, including layout management and file open or close operations, must use NFSv4.1 or NFSv4.2.