OneFS NFSv4.1 Trunking

As part of new OneFS 9.3 release’s support for NFSv4.1 and NFSv4.2, the NFS session model, is now incorporated into the OneFS NFS stack, which allows clients to leverage trunking and its associated performance benefits. Similar to multi-pathing in the SMB3 world, NFS trunking enables the use of multiple connections between a client and the cluster in order to dramatically increase the I/O path.

OneFS 9.3 supports both session and client ID trunking:

  • Client ID trunking is the association of multiple sessions per client.

  • Session trunking involves multiple connections per mount.

A connection, which represents a socket, exists within an object called a channel, and there can be many sessions associated with a channel. The fore channel represents client > cluster communication, and the back channel cluster > client.

Each channel has a set of configuration values that affect a session’s connections. With a few exceptions, the cluster must respect client-negotiated values. Typically, the configuration value meanings are the same for both the fore and back channels, although the defaults are typically significantly different for each.

Also, be aware that there can only be one client per session, but multiple sessions per client. And here’s what combined session and client ID trunking looks like:

Most Linux flavors support session trunking via the ‘nconnect’ option within the ‘mount’  command, which is included in kernel version 5.3 and later. However, support for client ID trunking is fairly nascent across the current Linux distributions. As such, we’ll focus on session trunking for the remainder of this article.

So let’s walk through a simple example of configuring NFS v4.1 and session trunking in OneFS 9.3.

The first step is to enable the NFS service, if it’s not already running, and select the desired protocol versions. This can be done from the CLI via the following command syntax:

# isi services nfs enable
# isi nfs settings global modify --nfsv41-enabled=true --nfsv42-enabled=true

Next, create an NFS export:

# isi nfs exports create --paths=/ifs/data

When using NFSv4.x, the domain name should be uniform across both the cluster and client(s). The NFSv4.x domain is presented as user@domain or group@domain pairs in ‘getattr’ and ‘setattr’ operations, for example. If the domain does not match, new and existing files will appear as owned by user ‘nobody user on the cluster.

The cluster’s NFSv4.x domain can be configured via the CLI using the ‘isi nfs settings zone modify’ command as follows:

# isi nfs settings zone modify --nfsv4-domain=nfs41test --zone=System

Once the cluster is configured, the next step is to prepare the NFSv4.1 client(s). As mentioned previously, Linux clients running the 5.3 kernel or later can use the nconnect mount option to configure session trunking.

Note that the current maximum limit of client-server connections opened by nconnect is 16. If unspecified, this value defaults to 1.

The following example uses an Ubuntu 21.04 client with the Linux 5.11 kernel version. The linux client will need to have the ‘nfs-common’ package installed in order to obtain the necessary nconnect binaries and libraries. If not already present, this can be installed as follows:

# sudo apt-get install nfs-common nfs-kernel-server

Next, edit the client’s /etc/idmapd.conf and add the appropriate the NFSv4.x domain:

# cat /etc/idmapd.conf

[General]

Verbosity = 0

Pipefs-Directory = /run/rpc_pipefs

# set your own domain here, if it differs from FQDN minus hostname

Domain = nfs41test

[Mapping]

Nobody-User = nobody

Nobody-Group = nogroup

NFSv4.x clients use the nfsidmap daemon for the NFSv4.x ID <-> name mapping translation, and the following CLI commands will restart the nfs-idmapd daemon and confirm that it’s happily running:

# systemctl restart nfs-idmapd
# systemctl status nfs-idmapd

 nfs-idmapd.service - NFSv4 ID-name mapping service

     Loaded: loaded (/lib/systemd/system/nfs-idmapd.service; static)

     Active: active (running) since Thurs 2021-11-18 19:47:01 PDT; 6s ago

    Process: 2611 ExecStart=/usr/sbin/rpc.idmapd $RPCIDMAPDARGS (code=exited, status=0/SUCCESS)

   Main PID: 2612 (rpc.idmapd)

      Tasks: 1 (limit: 4595)

     Memory: 316.0K

     CGroup: /system.slice/nfs-idmapd.service

             └─2612 /usr/sbin/rpc.idmapd

Nov 18 19:47:01 ubuntu systemd[1]: Starting NFSv4 ID-name mapping service...

Nov 18 25 19:47:01 ubuntu systemd[1]: Started NFSv4 ID-name mapping service.

The domain value can also be verified by running the nfsidmap command as follows:.

# sudo nfsidmap -d

nfs41test

Next, mount the cluster’s NFS export via NFSv4.1, v4.2, and trunking, as desired. For example, the following syntax will establish an NFSv4.1 mount using 4 trunked sessions, specified via the nconnect argument:

# sudo mount -t nfs -vo nfsvers=4.1,nconnect=4 10.1.128.10:/ifs/data/ /mnt/nfs41

This can be verified on the client side by running nestat and grepping for port 2049, the output in this case confirming the four TCP connections established for the above mount, as expected:

# netstat -ant4 | grep 2049

tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN    

tcp        0      0 10.1.128.131:857     10.1.128.10:2049     ESTABLISHED

tcp        0      0 10.1.128.131:681     10.1.128.10:2049     ESTABLISHED

tcp        0      0 10.1.128.131:738     10.1.128.10:2049     ESTABLISHED

tcp        0      0 10.1.128.131:959     10.1.128.10:2049     ESTABLISHED

Similarly, from the cluster side, the NFS connections can be checked with the OneFS ‘isi_nfs4mgmt’ CLI command. The command output includes the client ID, NFS version, session ID, etc.

# isi_nfs4mgmt –list

ID                 Vers  Conn  SessionId  Client Address  Port  O-Owners  Opens Handles L-Owners

456576977838751506  4.1   n/a   4          912.168.198.131 959   0         0     0       0

The OneFS isi_nfs4mgmt CLI command also includes a ‘—dump’ flag, which when used with the ID as the argument, will display the details of a client mount, such as the TCP port, NFSv4.1 channel options, auth type, etc.

# isi_nfs4mgmt --dump=456576977838751506

Dump of client 456576977838751506

  Open Owners (0):

Session ID: 4

Forward Channel

Connections:

             Remote: 10.1.128.131.959    Local: 10.1.128.10.2049

             Remote: 10.1.128.131.738    Local: 10.1.128.10.2049

             Remote: 10.1.128.131.857    Local: 10.1.128.10.2049

             Remote: 10.1.128.131.681    Local: 10.1.128.10.2049

Attributes:

             header pad size                  0

             max operations                   8

             max request size           1048576

             max requests                    64

             max response size          1048576

             max response size cached      7584


Slots Used/Available: 1/63

         Cache Contents:

             0)  SEQUENCE


Back Channel

Connections:

             Remote: 10.1.128.131.959    Local: 10.1.128.10.2049

Attributes:

             header pad size                  0

             max operations                   2

             max request size              4096

             max requests                    16

             max response size             4096

             max response size cached         0

Security Attributes:

         AUTH_SYS:

             gid                              0

             uid                              0


Summary of Client 456576977838751506:

  Long Name (hex): 0x4c696e7578204e465376342e31207562756e74752e312f3139322e3136382e3139382e313000

  Long Name (ascii): Linux.NFSv4.1.ubuntu.1/10.1.128.10.

  State: Confirmed

  Open Owners: 0

  Opens: 0

  Open Handles: 0

  Lock Owners: 0

  Sessions: 1

Full JSON dump can be found at /var/isi_nfs4mgmt/nfs_clients.dump_2021-11-18T15:25:18

Be aware that sessions trunking is not permitted across access zones, because of different auth levels, since a session represents a single auth level. Similarly, sessions trunking is disallowed across dynamic IP addresses.

Leave a Reply

Your email address will not be published.