OneFS NFS Locking and Reporting – Part 2

In the previous article in this series we took a look at the new NFS locks and waiters reporting CLI command set and API endpoints in OneFS 9.5 and later. Next, we turn our attention to some additional context, caveats, and NFSv3 lock removal.

Before to the NFS locking enhancements in OneFS 9.5, the legacy CLI commands were somewhat inefficient.  Their output also included other advisory domain locks, such as SMB, which made the output harder to parse. The table below maps the new 9.5 CLI commands (and corresponding handlers), to the old NLM syntax. Note that the ‘isi_classic nfs locks and waiters’ CLI commands have also been deprecated in OneFS 9.5.

Type / Command set OneFS 9.5 and later OneFS 9.4 and earlier
Locks isi nfs locks isi nfs nlm locks
Sessions isi nfs nlm sessions isi nfs nlm sessions
Waiters isi nfs locks waiters isi nfs nlm locks waiters

When upgrading to OneFS 9.5 or later from a prior release, the legacy platform API handlers continue to function through and post upgrade. So any legacy scripts and automation are protected from this lock reporting deprecation. Additionally, while the new platform API handlers will work in during a rolling upgrade in mixed-mode, they will only return results for the nodes that have already been upgdraded (‘high nodes’).

Be aware that the NFS locking CLI framework does not support partial responses. However, if a node is down or the cluster has a rolling upgrade in progress, the alternative is to query the equivalent platform API endpoint instead.

Performance-wise, on very large, busy clusters, there is the possibility that the lock and waiter CLI commands’ output will be sluggish. In such instances, the ’–timeout flag’ can be used to increase the command timeout window. Additionally, output filtering can also be used to reduce number of locks reported.

When a lock is in a transition state, there is a chance that it may not have/report a version. In these instances, the ‘Version’ field will be represented as ‘—‘. For example:

# isi nfs locks list -v

Client: 1/TMECLI1:487722/10.22.10.250

Client ID: 487722351064074

LIN: 4295164422

Path: /ifs/locks/nfsv3/10.22.10.250_1

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:03:52

Version: -

---------------------------------------------------------------

Total: 1

This behavior should be experienced very infrequently. However, if it is encountered, just execute the CLI command again and the lock version should be reported correctly.

When it comes to troubleshooting, NFSv3/NLM issues, if an NFSv3 client is consistently experiencing ‘NLM_DENIED’ or other lock management issues, this is often as a result of incorrectly configured firewall rules. For example, take the following packet capture (PCAP) excerpt from a Linux client:

   21 08:50:42.173300992  10.22.10.100 → 10.22.10.200 NLM 106    V4 LOCK Reply (Call In 19) NLM_DENIED

Often the assumption is that only the lockd or statd ports on the server side of the firewall need to be opened, and that the client always makes that connection that way. However, this is not the case. Instead, the server will frequently respond with a ‘let me get back to you’, then, later, it will reconnect to the client. As such, if the firewall blocks access to rcpbind on the client and/or lockd or statd on the client, connection failures will likely occur.

Occasionally it does become necessary to remove NLM locks and waiters from the cluster. Traditionally, the ‘isi_classic nfs clients rm’ command was frequently used, but that command has limitations, and is fully deprecated in OneFS 9.5 and later. Instead, the preferred method is to use the ‘isi nfs nlm sessions’ CLI utility, in conjunction with various other ancillary OneFS CLI commands, to clear problematic locks and waiters.

Note that the  ‘isi nfs nlm sessions’ CLI command, available in all current OneFS version, is Zone-Aware, and the output formatting is seen in the output for the client holding the lock, as it now shows the Zone ID number at the beginning. For example:

4/tme-linux1/10.22.10.250

The above represents:

Zone ID / Client _name / IP address of cluster node holding the connection.

A basic procedure to remove NFSv3 NLM locks and waiters from a cluster is as follows:

1.            First, list the NFS locks and search for the pertinent filename.

In OneFS 9.5 and later, the locks list can be filtered using the ‘–path’ argument.

# isi nfs locks list --path=<path> | grep <filename>

Be aware that the full path must be specified, starting with /ifs. There is no partial matching or substitution for paths in this command set.

For OneFS 9.4 and earlier, the following CLI syntax can be used:

#  isi_for_array -sX 'isi nfs nlm locks list | grep <filename>'

2.            Next, list the lock waiters associated with the same filename.

For OneFS 9.5 and later, the waiters list can also be filtered using the ‘–path’ syntax:

# isi nfs locks waiters –path=<path> | grep <filename>

With OneFS 9.4 and earlier, the following CLI syntax can be used:

# isi_for_array -sX 'isi nfs nlm locks waiters |grep -i <filename>'
  1. The next step is to confirm the client and LIN (logical inode number) being waited upon. This can be accomplished by querying the ‘efs.advlock.failover.lock_waiters’ sysctrl. For example:
# isi_for_array -sX 'sysctl efs.advlock.failover.lock_waiters'

For example:

[truncated output]
...
client = { '4/tme-linux1/10.20.10.200’, 0x26593d37370041 }
...
resource = 2:df86:0218

Note that for sanity checking, the ‘isi get -L’ CLI utility can be used to confirm the path of a file from its LIN:

# isi get -L <LIN>
  1.  The penultimate step involves the actual removal of the unwanted locks which are causing waiters to stack up.

Keep in mind that the ‘isi nfs nlm sessions’ command syntax is access zone-aware. As such, the following syntax will List the access zones by their IDs:

# isi zone zones list -v | grep -iE "Zone ID|name"

Once the desired zone ID has been determined, the ‘isi_run -z’ CLI utility can be used to specify the appropriate zone in which to run the ‘isi nfs nlm sessions’ commands:

# isi_run -z 4 -l root

Next, the ‘isi nfs nlm sessions delete’ CLI command will remove the specific lock waiter which is causing the issue. The command syntax requires specifying the client hostname and node IP of the node holding the lock.

# isi nfs nlm sessions delete –-zone <AZ_zone_ID> <hostname> <cluster-ip>

For example:

# isi nfs nlm sessions delete –zone 4 tme-linux1 10.20.10.200
Are you sure you want to delete all NFSv3 locks associated with client tme-linux1 against cluster IP 10.20.10.100? (yes/[no]): yes

5.   Finally, repeat the commands in step 1 above to confirm that the desired NLM locks and waiters have been successfully culled.

OneFS NFS Locking and Reporting

Also included among the plethora of OneFS 9.5 enhancements is an updated NFS lock reporting infrastructure, command set, and corresponding platform API endpoints. This new functionality includes enhanced listing and filtering options for both locks and waiters, based on NFS major version, client, LIN, path, creation time, etc. But first, some backstory…

The ubiquitous NFS protocol underwent some fundamental architectural changes between its versions 3 and 4. One of the major differences concerns the area of file locking.

NFSv4 is the most current major version of the protocol, and it natively incorporates file locking, thereby avoiding the need for any additional (and convoluted) RPC callback mechanisms, as with prior NFS versions. With NFSv4, locking is built in the main file protocol and supports new lock types, such as range locks, share reservations, and delegations/oplocks, which emulate those found in Window and SMB.

File lock state is maintained at the server under a lease-based model. A server defines a single lease period for all states held by an NFS client. If the client does not renew its lease within the defined period, all state associated with the client’s lease may be released by the server. If released, the client may either explicitly renew its lease, or simply issue a read request, or other associated operation. Additionally, with NFSv4, a client can elect whether to lock the entire file, or a byte range within a file.

In contrast to NFSv4, the NFSv3 protocol is stateless and does not natively support file locking. Instead, the ancillary NLM (Network Lock Manager) protocol supplies the locking layer. Because file locking is inherently stateful, NLM itself is considered stateful. For example, when an NFSv3 filesystem mounted on an NFS client receives a request to lock a file, instead of an NFS remote procedure call, it generates an NLM remote procedure call.

The NLM protocol itself consists of remote procedure calls that emulate the standard UNIX file control (fcntl) arguments and outputs. Because a process blocks waiting for a lock that conflicts with another lock holder, also known as a ‘blocking lock’, the NLM protocol has the notion of callbacks, from the file server to the NLM client to notify that a lock is available. As such, the NLM client sometimes acts as an RPC server in order to receive delayed results from lock calls.

Attribute NFSv3 NFSv4
State Stateless – A client does not technically establish a new session if it has the correct information to ask for files and so on. This allows for simple failover between OneFS nodes using dynamic IP pools Stateful – NFSv4 uses sessions in order to handle communication, as such both client and server must track session state to continue communicating.
Presentation User and Group info is presented numerically – Client and Server communicate user information by numeric identifiers, allowing the same user to appear as different names between client and server. User and Group info is presented as strings – Both the client and server must resolve the names of the numeric information stored. The server must lookup names to present, while the client must remap those to numbers on its end
Locking File Locking is out of band – uses NLM to perform locks. This requires the client to respond to RPC messages from the server to confirm locks have been granted, etc. File Locking is in band – No longer users a separate protocol for file locking, instead making it a type of call that is usually compounded with OPENs, CREATES, or WRITES
Transport Can run over TCP or UDP – This version of the protocol can run over UDP instead of TCP, leaving handling of loss and retransmission to the software instead of the operating system. We always recommend using TCP. Only supports TCP – Version 4 of NFS has left loss and retransmission up to the underlying operating system.

Can batch a series of calls in a single packet, allowing the server to process all of them and reply at the end. This is used to reduce the number of calls involved in common operations.

Since NFSv3 is stateless it requires more complexity to recover from failures such as client and server outages and network partitions. If an NLM server crashes, NLM clients that are holding locks must reestablish them on the server when it restarts. The NLM protocol deals with this by having the status monitor on the server send a notification message to the status monitor of each NLM client that was holding locks. The initial period after a server restart is known as the grace period, during which only requests to reestablish locks are granted. Thus, clients that reestablish locks during the grace period are guaranteed to not lose their locks.

When an NLM client crashes, ideally any locks it was holding at the time are removed from the pertinent NLM server(s). The NLM protocol handles this by having the status monitor on the client send a message to each server’s status monitor once the client reboots. The client reboot indication informs the server that the client no longer requires its locks. However, if the client crashes and fails to reboot, the client’s locks will persist indefinitely. This is undesirable for two primary reasons: Resources are indefinitely leaked. Eventually another client will want to get a conflicting lock on at least one of the files the crashed client had locked and, as a result, the other client is postponed indefinitely.

Therefore, having NFS server utilities to swiftly and accurately report on lock and waiter status, and utilities to clear NFS lock waiters is highly desirable for administrators – particularly on clustered storage architectures.

Prior to OneFS 9.5, the old NFS locking CLI commands were somewhat inefficient and also showed other advisory domain locks, which rendered the output somewhat confusing. The following table shows the new CLI commands (and corresponding handlers), which replace the older NLM syntax.

Type / Command set OneFS 9.4 and earlier OneFS 9.5
Locks isi nfs nlm locks isi nfs locks
Sessions isi nfs nlm sessions isi nfs nlm sessions
Waiters isi nfs nlm locks waiters isi nfs locks waiters

In OneFS 9.5 and later, the old API handlers will still exist to avoid breaking existing scripts and automation, but the CLI command syntax is deprecated and will no longer work.

Also be aware that the ‘isi_classic nfs locks and waiters’ CLI commands have also been disabled in OneFS 9.5. Attempts to run these will yield the following warning message:

# isi_classic nfs locks

This command has been disabled. Please use 'isi nfs' for this functionality.

The new ‘isi nfs locks’ CLI command output includes the following locks object fields:

Field Description
Client The client host name, Frequently Qualified Domain Name, or IP
Client_ID The client ID (internally generated)
Created The UNIX Epoch time that the lock was created
ID The lock ID (Id necessary for platform API sorting, not shown in CLI output)
LIN The logical inode number (LIN) of the locked resource
Lock_type The type of lock (shared, exclusive, none)
Path Path of locked file
Range The byte range within the file that is locked
Version The NFS major version: v3, or v4

Note that the ISI_NFS_PRIV RBAC privilege is required in order to view the NFS locks or waiters via the CLI or PAPI. In addition to ‘root’, the cluster’s ‘SystemAdmin’ and ‘SecurityAdmin’ roles contain this privilege by default.

Additionally, the new locks CLI command sets ha a default timeout of 60 seconds. If the cluster is very large, the timeout may need to be increased for the CLI command. For example:

# isi –timeout <timeout value> nfs locks list

The basic architecture behind the enhanced NFS locks reporting framework is as follows:

The new API handlers leverage the platform API proxy, yielding increased performance over the legacy handlers. Additionally, updated syscalls have been implemented to facilitate filtering by NFS service and major version.

Since NFSv3 is stateless, the cluster does not know when a client has lost its state unless it reconnects. For maximum safety, the OneFS locking framework (lk) holds locks forever. The ‘isi nfs nlm sessions’ CLI command allows administrators to manually free NFSv3 locks in such cases, and this command remains available in OneFS 9.5 as well as prior versions. NFSv3 locks may also be leaked on delete, since a valid inode is required for lock operations. As such, lkf has a lock reaper which periodically checks for locks associated with deleted files.

In OneFS 9.5 and later, current NFS locks can be viewed with the new ‘isi nfs locks list’ command. This command set also provides a variety of options to limit and format the display output. In its basic form, this command generates a basic list of client IP address and the path. For example:

# isi nfs locks list

Client                              Path

-------------------------------------------------------------------

1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_1

1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_2

Linux NFSv4.0 TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv4/10.22.10.250_1

Linux NFSv4.0 TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv4/10.22.10.250_2

-------------------------------------------------------------------

Total: 4

To include more information, the ‘-v’ flag can be used to generate a verbose locks listing:

 # isi nfs locks list -v

Client: 1/TMECLI1:487722/10.22.10.250

Client ID: 487722351064074

LIN: 4295164422

Path: /ifs/locks/nfsv3/10.22.10.250_1

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:03:52

Version: v3

---------------------------------------------------------------

Client: 1/TMECLI1:487722/10.22.10.250

Client ID: 5175867327774721

LIN: 42950335042

Path: /ifs/locks/nfsv3/10.22.10.250_1

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:10:31

Version: v3

---------------------------------------------------------------

Client: Linux NFSv4.0 TMECLI1:487722/10.22.10.250

Client ID: 487722351064074

LIN: 429516442

Path: /ifs/locks/nfsv3/10.22.10.250_1

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:19:48

Version: v4

---------------------------------------------------------------

Client: Linux NFSv4.0 TMECLI1:487722/10.22.10.250

Client ID: 487722351064074

LIN: 4295426674

Path: /ifs/locks/nfsv3/10.22.10.250_2

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:17:02

Version: v4

---------------------------------------------------------------

Total: 4

The above syntax returns more detailed information for each lock, including client ID, LIN, path, lock type, range, created date, and NFS version.

The lock listings can also be filtered by client or client-id. Note that the –client option must be the full name in quotes.

# isi nfs locks list --client="full_name_of_client/IP_address" -v

For example:

# isi nfs locks list --client="1/TMECLI1:487722/10.22.10.250" -v

Client: 1/TMECLI1:487722/10.22.10.250

Client ID: 5175867327774721

LIN: 42950335042

Path: /ifs/locks/nfsv3/10.22.10.250_1

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:10:31

Version: v3

Additionally, be aware that the CLI does not support partial names, so the full name of the client must be specified.

Filtering by NFS version can be helpful when attempting to narrow down which client has a lock. For example, to show just the NFSv3 locks:

# isi nfs locks list --version=v3

Client                              Path

-------------------------------------------------------------------

1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_1

1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_2

-------------------------------------------------------------------

Total: 2

Note that the ‘–-version’ flag supports both ‘v3’ and ‘nlm’ as arguments, and will return the same v3 output in either case. For example:

# isi nfs locks list --version=nlm

Client                              Path

-------------------------------------------------------------------

1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_1

1/TMECLI1:487722/10.22.10.250       /ifs/locks/nfsv3/10.22.10.250_2

-------------------------------------------------------------------

Total: 2

Filtering  by LIN or path is also supported. For example:

# isi nfs locks list --lin=42950335042 -v

Client: 1/TMECLI1:487722/10.22.10.250

Client ID: 5175867327774721

LIN: 42950335042

Path: /ifs/locks/nfsv3/10.22.10.250_1

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:10:31

Version: v3

Or by path:

# isi nfs locks list --path=/ifs/locks/nfsv3/10.22.10.250_2

 -v

Client: Linux NFSv4.0 TMECLI1:487722/10.22.10.250

Client ID: 487722351064074

LIN: 4295426674

Path: /ifs/locks/nfsv3/10.22.10.250_2

Lock Type: exclusive

Range: 0, 92233772036854775807

Created: 2023-08-18T08:17:02

Version: v4

Be aware that the full path must be specified, starting with /ifs. There is no partial matching or substitution for paths in this command set.

Filtering can also be performed by creation time, for example:

# isi nfs locks list --created=2023-08-17T09:30:00 -v

Note that, when filtering by ‘created’ the output will include all locks that were created before or at the time provided.

The ‘—limits’ argument can be used curtail the number of results returned, and ‘limits’ can be used in conjunction with all other query options. For example, to limit the output of the NFSv4 locks listing to one lock:

# isi nfs locks list -–version=v4 --limit=1

Note that ‘limit’ can be used with the range of query types.

The filter options are mutually exclusive with the exception of ‘version’, and ‘version’ can be used with any of the other filter options. For example, filtering by both ‘created’ and ‘version’:

This can be helpful when troubleshooting and trying to narrow down results.

In addition to locks, OneFS 9.5 also provides the ‘isi nfs locks waiters’ CLI command set. Note that ‘waiters’ are specific to NFSv3 clients, and the CLI reports any v3 locks that are pending and not yet granted.

Since NFSv3 is stateless, a cluster does not know when a client has lost its state unless it reconnects. For maximum safety, lk holds locks forever. The isi nfs nlm command allows administrators to manually free locks in such cases. Locks may also be leaked on delete, since a valid inode is required for lock operations. Thus lkf has a lock reaper which periodically checks for locks associated with deleted files

# isi nfs locks waiters

The ‘waiters’ CLI syntax uses a similar range of query arguments as the ‘isi nfs locks list’ command set.

In addition to the CLI, the platform API can also be used to query both NFS locks and NFSv3 waiters. For example, using ‘curl’ to view the waiters via the OneFS pAPI:

# curl -k -u <username>:<passwd> https://localhost:8080/platform/protocols/nfs/waiters”

{

“total” : 2,

“waiters”;

}

{

“client” : “1/TMECLI1487722/10.22.10.250”,

“client_id” : “4894369235106074”,

“created” : “1668146840”,

“id” : “1 1YUIAEIHVDGghSCHGRFHTiytr3u243567klj212-MANJKJHTTy1u23434yui-ouih23ui4yusdftyuySTDGJSDHVHGDRFhgfu234447g4bZHXhiuhsdm”,

“lin” : “4295164422”,

“lock_type” : “exclusive”

“path” : “/ifs/locks/nfsv3/10.22.10.250_1”

“range” : [0, 92233772036854775807 ],

“version” : “v3”

}

},

“total” : 1

}

Similarly, using the platform API to show locks, filtered by client ID:

# curl -k -u <username>:<passwd> “https://<address>:8080/platform/protocols/nfs/locks?client=<client_ID>”

For example:

# curl -k -u <username>:<passwd> “https://localhost:8080/platform/protocols/nfs/locks?client=1/TMECLI1487722/10.22.10.250”

{

“locks”;

}

{

“client” : “1/TMECLI1487722/10.22.10.250”,

“client_id” : “487722351064074”,

“created” : “1668146840”,

“id” : “1 1YUIAEIHVDGghSCHGRFHTiytr3u243567FCUJHBKD34NMDagNLKYGHKHGKjhklj212-MANJKJHTTy1u23434yui-ouih23ui4yusdftyuySTDGJSDHVHGDRFhgfu234447g4bZHXhiuhsdm”,

“lin” : “4295164422”,

“lock_type” : “exclusive”

“path” : “/ifs/locks/nfsv3/10.22.10.250_1”

“range” : [0, 92233772036854775807 ],

“version” : “v3”

}

},

“Total” : 1

}

Note that, as with the CLI, the platform API does not support partial name matches, so the full name of the client must be specified.

OneFS SSL Certificate Creation and Renewal – Part 2

In the initial article in this series, we took a look at the OneFS SSL architecture, plus the first two steps in the basic certificate renewal or creation flow:

The procedure below includes options to complete a self-signed certificate replacement or renewal, or to request an SSL replacement or renewal from a Certificate Authority (CA).

 

Signing the SSL Certificate

At this point, depending on the security requirements of the environment, the certificate can either be self-signed or signed by a Certificate Authority.

a.      Self-Sign the SSL Certificate.

The following CLI syntax can be used to self-sign the certificate with the key, which creates a new signed certificate which, in this instance, is valid for 1 year (365 days):

# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

To verify that the key matches the certificate, ensure that the output of the following CLI commands return the same md5 checksum value:

# openssl x509 -noout -modulus -in server.crt | openssl md5           
# openssl rsa -noout -modulus -in server.key | openssl md5

Next, proceed to the ‘Add certificate to cluster’ section of this article once this step is complete.

b.      Use a CA to Sign the Certificate.

If a CA is signing the certificate, ensure that the new SSL certificate is in x509 format, and includes the entire certificate trust chain.

Note that the CA may return the new SSL certificate, the intermediate cert, and the root cert in different files. If this is the case, the PEM formatted certificate will need to be created manually.

Also, the correct ordering is important when creating the PEM-formatted certificate. The SSL cert must be the top of the file, followed by the intermediate certificate, and with the root certificate at the bottom. For example:

-----BEGIN CERTIFICATE-----

<Contents of new SSL certificate>

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

<Contents of intermediate certificate>

<Repeat as necessary for every intermediate certificate provided by your CA>

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

<Contents of root certificate file>

-----END CERTIFICATE-----

A simple method for creating the PEM formatted file from the CLI is to ‘cat’ the in the correct order as follows:

# cat CA_signed.crt intermediate.crt root.crt > onefs_pem_formatted.crt

Copy the onefs_pem_formatted.crt file to /ifs/tmp and rename it to server.crt.

Note that if any of the above files are generated with a ‘.cer’ extension, it should be renamed with a .crt extension instead.

The attributes and integrity of the certificate can be sanity checked with the following CLI syntax:

# openssl x509 -text -noout -in server.crt

 

Adding the certificate to the cluster.    

The first step in adding the certificate involves importing the new certificate and key into the cluster:

# isi certificate server import /ifs/tmp/server.crt /ifs/tmp/server.key

Next, verify that the certificate imported successfully:

# isi certificate server list -v

Note that the following CLI command can be used to show just the names and corresponding IDs of the certificates:

# isi certificate server list -v | grep -A1 "ID:"

Set the freshly imported certificate to be the default:

# isi certificate settings modify --default-https-certificate=<id_of_cert_to_set_as_default>

Confirm that the imported certificate is being used as default by verifying status of ‘Default HTTPS Certificate’:

# isi certificate settings view

If there is an unused or outdated certificate, it can be deleted with the following CLI syntax:

# isi certificate server delete --id=<id_of_cert_to_delete>

Next, the new imported certificate can be viewed and verified with the following CLI command:

# isi certificate server view --id=<id_of_cert>

Note that ports 8081 and 8083 still use the certificate from the local directory for SSL. Follow the below steps if you want to use the new certificates for port 8081/8083.

# isi services -a isi_webui disable# chmod 640 server.key# chmod 640 server.crt# isi_for_array -s 'cp /ifs/tmp/server.key /usr/local/apache2/conf/ssl.key/server.key'# isi_for_array -s 'cp /ifs/tmp/server.crt /usr/local/apache2/conf/ssl.crt/server.crt'isi services -a isi_webui enable

 

Verifying the SSL certificate.

There are two simple ways of verifying the updated SSL certificate: Either via the CLI, using the ‘openssl’ command as follows:

# echo QUIT | openssl s_client -connect localhost:8080

Or via a web browser, using the following URL:

 https://<cluster_name>:8080

Note that where <cluster_name> is the FQDN or IP address that’s typically used to access the cluster’s WebUI interface. The security details for the web page will contain the location and contact info, as above.

In both cases, the output includes location and contact info. For example:

Subject: C=US, ST=<yourstate>, L=<yourcity>, O=<yourcompany>, CN=isilon.example.com/emailAddress=tme@isilon.com

Additionally, OneFS provides warning of an impending certificate expiry by sending a CELOG event alert along the lines of the following:

SW_CERTIFICATE_EXPIRING: X.509 certificate default is nearing expiration: 

Event: 400170001
Certificate 'default' in '**' store is nearing expiration:

Note that OneFS does not attempt to automatically renew a certificate. Instead an expiring cert has to be renewed manually, per the procedure described above.

When adding an additional certificate, any time you connect to that SmartConnect name via HTTPS, the matching cert is used. If no matching certificate is found, OneFS will automatically revert to using the default self-signed certificate.

OneFS SSL Certificate Renewal – Part 1

When using either the OneFS WebUI or platform API (pAPI), all communication sessions are encrypted using SSL (Secure Sockets Layer), also known as Transport Layer Security (TLS). In this series, we’ll take a look at how to replace or renew the SSL certificate for the OneFS WebUI.

SSL requires a certificate that serves two principal function: It grants permissions to use encrypted communication via Public Key Infrastructure, and also authenticates the identity of the certificate’s holder.

Architecturally, SSL comprises four fundamental components:

SSL Component Description
Alert Reports issues.
Change cipher spec Implements negotiated crypto parameters.
Handshake Negotiates crypto parameters for SSL session. Can be used for many SSL/TCP connections.
Record Provides encryption and MAC.

These sit in the stack as follows:

The basic handshake process begins with a client requesting an HTTPS WebUI session to the cluster. OneFS then returns the SSL certificate and public key. The client creates a session key, encrypted with the public key it’s received from OneFS. At this point, the client only knows the session key. The client now sends its encrypted session key to the cluster, which decrypts it with the private key. Now, both the client and OneFS know the session key. So, finally the session, encrypted via symmetric session key, can be established. OneFS automatically defaults to the best supported version of SSL, based on the client request.

A PowerScale cluster initially contains a self-signed certificate, which can be used as-is, or replaced with a third-party certificate authority (CA)-issued certificate. If the self-signed certificate is used, upon expiry, it must be replaced with either a third-party (public or private) CA-issued certificate or another self-signed certificate that is generated on the cluster. The following are the default locations for the server.crt and server.key files.

File Location
SSL certificate /usr/local/apache2/conf/ssl.crt/server.crt
SSL certificate key /usr/local/apache2/conf/ssl.key/server.key

The ‘isi certificate settings view’ CLI command displays all of the certificate-related configuration options. For example:

# isi certificate settings view

         Certificate Monitor Enabled: Yes

Certificate Pre Expiration Threshold: 4W2D

           Default HTTPS Certificate

                                      ID: default

                                 Subject: C=US, ST=Washington, L=Seattle, O="Isilon", OU=Isilon, CN=Dell, emailAddress=tme@isilon.com

                                  Status: valid

The above ‘certificate monitor enabled’ and ‘certificate pre expiration threshold’ config options govern a nightly cron job, which monitors the expiration of each managed certificate and fires a CELOG alert if a certificate is set to expire within the configured threshold. Note that the default expiration is 30 days (4W2D, which represents 4 weeks plus 2 days). The ‘ID: default’ config option indicates that this certificate is the default TLS certificate.

The basic certificate renewal or creation flow is as follows:

The steps below include options to complete a self-signed certificate replacement or renewal, or to request an SSL replacement or renewal from a Certificate Authority (CA).

Backing up the existing SSL certificate

The first task is to obtain the list of certificates by running the following CLI command, and identify the appropriate one to renew:

# isi certificate server list

ID      Name    Status  Expires

-------------------------------------------

eb0703b default valid   2025-10-11T10:45:52

-------------------------------------------

It’s always a prudent practice to save a backup of the original certificate and key. This can be easily accomplished via the following CLI commands, which, in this case, create the directory ‘/ifs/data/ssl_bkup’ directory, set the perms to root-only access, and copy the original key and certificate to it:

# mkdir -p /ifs/data/ssl_bkup

# chmod 700 /ifs/data/ssl_bkup

# cp /usr/local/apache24/conf/ssl.crt/server.crt /ifs/data/ssl_bkup

# cp /usr/local/apache24/conf/ssl.key/server.key /ifs/data/ssl_bkup

# cd !$

cd /ifs/data/ssl_bkup

# ls

server.crt      server.key

Renewing or creating a certificate

The next step in the process involves either the renewal of an existing certificate or creation of a certificate from scratch. In either case, first, create a temporary directory, for example /ifs/tmp:

# mkdir /ifs/tmp; cd /ifs/tmp

a)       Renew an existing self-signed Certificate.

The following syntax creates a renewal certificate that is based on the existing ssl.keyThe value of the ‘-days’ parameter can be adjusted to generate a certificate with the desired expiration date. For example, the following command will create a one-year certificate.

# cp /usr/local/apache2/conf/ssl.key/server.key ./ ; openssl req -new -days 365 -nodes -x509 -key server.key -out server.crt

Answer the system prompts to complete the self-signed SSL certificate generation process, entering the pertinent information location and contact information. For example:

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Washington
Locality Name (eg, city) []:Seattle
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Isilon
Organizational Unit Name (eg, section) []:TME
Common Name (e.g. server FQDN or YOUR name) []:isilon.com
Email Address []:tme@isilon.com

When all the information has been successfully entered, the server.csr and server.key files will be generated under the /ifs/tmp directory.

Optionally,  the attributes and integrity of the certificate can be verified with the following syntax:

# openssl x509 -text -noout -in server.crt

Next, proceed directly to the ‘Add the certificate to the cluster’ steps in section 4 of this article.

b)      Alternatively, a certificate and key can be generated from scratch, if preferred.

The following CLI command can be used to create an 2048-bit RSA private key:

# openssl genrsa -out server.key 2048

Generating RSA private key, 2048 bit long modulus

............+++++

...........................................................+++++

e is 65537 (0x10001)

Next, create a certificate signing request:

# openssl req -new -nodes -key server.key -out server.csr

For example:

# openssl req -new -nodes -key server.key -out server.csr -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:isilon.com"))

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [AU]:US

State or Province Name (full name) [Some-State]:WA

Locality Name (eg, city) []:Seattle

Organization Name (eg, company) [Internet Widgits Pty Ltd]:Isilon

Organizational Unit Name (eg, section) []:TME

Common Name (e.g. server FQDN or YOUR name) []:h7001

Email Address []:tme@isilon.com

Please enter the following 'extra' attributes

to be sent with your certificate request

A challenge password []:1234

An optional company name []:

#

Answer the system prompts to complete the self-signed SSL certificate generation process, entering the pertinent information location and contact information. Additionally, a ‘challenge password’ with a minimum of 4-bytes in length will need to be selected and entered.

As prompted, enter the information to be incorporated into the certificate request. When completed, the server.csr and server.key files will appear in the /ifs/tmp directory.

If desired, a CSR file for a Certificate Authority which includes Subject-Alternative-Names (SAN) can be generated. For example, additional host name entries can be added by using a comma (IE. DNS:isilon.com,DNS:www.isilon.com).

In the next article, we’ll look at the certificate singing, addition, and verification steps of the process.

OneFS SMB Redirector Encryption

As on-the-wire encryption becomes increasingly commonplace, and often mandated via regulatory compliance security requirements, the policies applied in enterprise networks are rapidly shifting towards fully encrypting all traffic.

The OneFS SMB protocol implementation (lwio) has supported encryption for Windows and other SMB client connections to a PowerScale cluster since OneFS 8.1.1, as detailed in this article.

However, prior to OneFS 9.5, this did not include encrypted communications between the SMB redirector and Active Directory (AD) domain controller (DC). While Microsoft added support for SMB encryption in SMB 3.0, the redirector in OneFS 9.4 and prior releases only supported Microsoft’s earlier SMB 2.002 dialect.

When OneFS connects to Active Directory for tasks requiring remote procedure calls (RPCs), such as joining a domain, NTLM authentication, or resolving usernames and SIDs, these SMB connections are established from OneFS as the client connecting to a domain controller server.

As outlined in the Windows SMB security documentation, by default, and starting with Windows 2012 R2, domain admins can choose to encrypt access to a file share, which can include a domain controller. When encryption is enabled, only SMB3 connections are permitted.

With OneFS 9.5, the OneFS SMB redirector now supports SMB3, thereby allowing the Local Security Authority Subsystem Service (LSASS) daemon to communicate with domain controllers running Windows Server 2012 R2 and later over an encrypted session.

The OneFS redirector, also known as the ‘rdr driver’, is a stripped-down SMB client with minimal functionality, only supporting what is absolutely necessary.

Under the hood, OneFS SMB encryption and decryption use standard OpenSSL functions, and AES-128-CCM encryption is negotiated during SMB negotiate phase.

Although everything stems from the NTLM authentication requested by SMB server, the sequence of calls leads to the redirector establishing an SMB connection to the AD domain controller.

With OneFS 9.5, no configuration is required to enable SMB encryption in most situations, and there are no WebUI OR CLI configuration settings for the redirector.

With the default OneFS configuration, the redirector supports encryption if negotiated but it does not require it. Similarly, if the Active Directory domain requires encryption, the OneFS redirector will automatically enable and use encryption. However, if the OneFS redirector is explicitly configured to require encryption and the domain controller does not support encryption, the connection will fail.

The OneFS redirector encryption settings include:

Key Values Description
Smb3EncryptionEnabled Boolean. Default is ‘1’ == Enabled Enable or disable SMB3 encryption for OneFS redirector.
Smb3EncryptionRequired Boolean. Default is ‘0’ == Not required. Require or not require redirector connection to be encrypted.
MaxSmb2DialectVersion Default is ‘max’ == SMB 3.0.2 Set the SMB dialect the redirector will support.  Maximum is currently SMB 3.0.2.

The above keys and values are stored in the OneFS Likewise SMB registry and can be viewed and configured with the ‘lwreqshell’ utility. For example, to view the SMB redirector encryption config settings:

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i encrypt

   "Smb3EncryptionEnabled"  REG_DWORD       0x00000001 (1)

   "Smb3EncryptionRequired" REG_DWORD       0x00000000 (0)

The following syntax can be used to disable the ‘Smb3EncryptionRequired’ parameter by setting it to value ‘1’:

# /usr/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]" "Smb3EncryptionRequired" "0x00000001"

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i encrypt

   "Smb3EncryptionEnabled"  REG_DWORD       0x00000001 (1)

   "Smb3EncryptionRequired" REG_DWORD       0x00000001 (1)

Similarly, to restore the ‘Smb3EncryptionRequired’ parameter’s default value of ‘0’ (ie. not required):

# /usr/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr]" "Smb3EncryptionEnabled" "0x00000001"

Note that, during the upgrade to OneFS 9.5, any nodes still running the old version will not be able to NTLM-authenticate if the DC they have affinity with requires encryption.

While redirector encryption is implemented in user space (in contrast to the SMB server, which is in the kernel), since  it involves OpenSSL, the library does take advantage of hardware acceleration on the processor and utilizes AES-NI. As such, performance is only minimally impacted when the number of NTLM authentications to the AD domain is very large.

Also note that redirector encryption also only currently supports only AES-128-CCM encryption provided in the SMB 3.0.0 and 3.0.2 dialects. OneFS does not use AES-128-GCM encryption, available in the SMB 3.1.1 dialect (the latest), at this time.

When it comes to troubleshooting the redirector, the lwregshell tool can be used to verify its configuration settings. For example, to view the redirector encryption settings:

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i encrypt

   "Smb3EncryptionEnabled"  REG_DWORD       0x00000001 (1)

   "Smb3EncryptionRequired" REG_DWORD       0x00000000 (0)

Similarly, to find the maximum SMB version supported by the redirector:

# /usr/likewise/bin/lwregshell list_values "HKEY_THIS_MACHINE\Services\lwio\Parameters\Drivers\rdr" | grep -i dialect

   "MaxSmb2DialectVersion"  REG_SZ          "max"

The ‘lwsm’ CLI utility with the following syntax will confirm the status of the various lsass components:

# /usr/likewise/bin/lwsm list | grep lsass

lsass                      [service]     running (lsass: 5164)

netlogon                   [service]     running (lsass: 5164)

rdr                        [driver]      running (lsass: 5164)

It can also be used to show and modify the logging level. For example:

# /usr/likewise/bin/lwsm get-log rdr

<default>: syslog LOG_CIFS at WARNING

# /usr/likewise/bin/lwsm set-log-level rdr - debug

# /usr/likewise/bin/lwsm get-log rdr

<default>: syslog LOG_CIFS at DEBUG

When finished, rdr logging can be returned to its previous log level as follows:

# /usr/likewise/bin/lwsm set-log-level rdr - warning

# /usr/likewise/bin/lwsm get-log rdr

<default>: syslog LOG_CIFS at WARNING

Additionally, the existing ‘lwio-tool’ utility has been modified in OneFS 9.5 to include functionality allowing simple test connections to domain controllers (no NTLM) via the new ‘rdr’ syntax:

# /usr/likewise/bin/lwio-tool rdr openpipe //<domain_controller>/NETLOGON

The ‘lwio-tool’ usage in OneFS 9.5 is as follows:

# /usr/likewise/bin/lwio-tool -h

Usage: lwio-tool <command> [command-args]

  commands:

    iotest rundown

    rdr [openpipe|openfile] username@password://domain/path

    srvtest transport [query|start|stop]

    testfileapi [create|createnp] <path>

OneFS WebUI Single Sign-on Management and Troubleshooting

Earlier in this series, we took a look at the architecture of the new OneFS WebUI SSO functionality. Now, we move on to its management and troubleshooting.

As we saw in the previous article, once the IdP and SP are configured, a cluster admin can enable SSO per access zone via the OneFS WebUI by navigating to Access > Authentication providers > SSO. From here, select the desired access zone and click the ‘Enable SSO’ toggle:

Or from the OneFS CLI via the following syntax:

# isi auth sso settings modify –sso-enabled 1

Once complete, the SSO configuration can be verified from a client web browser by browsing to the OneFS login screen. If all is operating correctly, redirection to the ADFS login screen will occurs. For example:

After successful authentication with ADFS, cluster access is granted and the browser session is redirected back to the OneFS WebUI .

In addition to the new SSO WebUI pages, OneFS 9.5 also adds a sub-command to the ‘isi auth’ command set for configuring SSO from the CLI. This new syntax includes:

  • isi auth sso idps
  • isi auth sso settings
  • isi auth sso sp

With these, the following procedure can be used to configure and enable SSO via the OneFS CLI.

  1. Define the ADFS instance in OneFS.

Create the IdP.

# isi auth ads create <domain_name> <user> --password=<password> ...

Where:

Attribute Description
<domain_name> Fully qualified Active Directory domain name that identifies the ADFS server. For example, idp1.isilon.com.
<user> The user account with permission to join machines to the given domain.
<password> The password for <user>.

 

  1. Add the Active Directory IdP to OneFS zones. Note that each of a cluster’s access zone(s) must have an IdP configured for it. The same IdP can be used for all the zones, but each access zone must be configured separately.
# isi zone zones modify --add-auth-providers

For example:

# isi zone zones modify system –add-auth-providers=lsa-activedirectoryprovider:idp1.isilon.com

  1. Verify that OneFS can find users in Active Directory.
# isi auth users view idp1.isilon.com\\<username>

In the output, ensure that an email address is displayed. If not, return to Active Directory and assign email addresses to users.

  1. Configure the OneFS hostname for SAML SSO.
# isi auth sso sp modify --hostname=<name>

Where <name> is the name that SAML SSO can use to represent the OneFS cluster to ADFS. SAML redirects clients to this hostname.

  1. Obtain the ADFS metadata and store it on  the cluster.

In the following example, an HTTPS GET request is issued via the ‘curl’ utility to obtain the metadata from the IDP and store it under /ifs on the cluster.

# curl -o /ifs/adfs.xml https://idp1.isilon.com/FederationMetadata/2008-07/ FederationMetadata.xml
  1. Create the IdP on OneFS using the ‘metadata-location’ path for the xml file in the previous step.
# isi auth sso idps create idp1.isilon.com --metadata-location="/ifs/adfs.xml"
  1. Enable SSO:
# isi auth sso settings modify --sso-enabled=yes -–zone <zone>

The following syntax can be used to view the IdP configuration:

# isi auth sso idps view <idp_ID>

For example:

# isi auth sso idps view idp

ID: idp

Metadata Location: /ifs/adfs.xml

Entity ID: https://dns.isilon.com/adfs/services/trust

Login endpoint of the IDP

URL: https://dns.isilon.com/adfs/ls/

Binding: wrn:oasis:names:tc:SAML:2.0:bidings:HTTP-Redirect

Logout endpoint of the IDP

URL: https://dns.isilon.com/adfs/ls/

Binding: wrn:oasis:names:tc:SAML:2.0:bidings:HTTP-Redirect

Response URL: -

Type: metadata

Signing Certificate: -

      Path:

      Issuer: CN-ADFS Signing – dns.isilon.com

      Subject: CN-ADFS Signing – dns.isilon.com

      Not Before: 2023-02-02T22:22:00

      Not After: 2024-02-02T22:22:00

      Status: valid

Value and Type

      Value: -----BEGIN CERTIFICATE-----

MITC9DCCAdygAwIBAgIQQQQc55appr1CtfPNj5kv+DANBgkqhk1G9w8BAQsFADA2

<snip>

Troubleshooting.

In the event that the IdP and/or SP Signing certificate happen to expire, users will be unable to login to the cluster via SSO and an error message will be displayed on login page.

In the example above, the IdP certificate has expired, as described in the alert message. In this situation, a warning is also displayed on the SSO Authentication page, as shown below:

To correct this, download either a new signing certificate from the Identity Provider or a new metadata file containing the IdP certificate details. Once complete, the cluster’s IdP configuration can then be updated by uploading the XML file or the new certificate.

Similarly, if the SP certificate has expired, the following notification alert is displayed upon attempted login:

The error message below is also displayed on the WebUI SSO tab, under Access > Authentication providers > SSO, along with a link to regenerate the metadata file:

The expired SP signing key and certificate can also be easily regenerated from the OneFS CLI:

# isi auth sso sp signing-key rekey

This command will delete any existing signing key and certificate and replace them with a newly generated signing key and certificate. Make sure the newly generated certificate is added to the IDP to ensure that the IDP can verify messages sent from the cluster. Are you sure?  (yes/[no]):  yes

# isi auth sso sp signing-key dump

-----BEGIN CERIFICATE-----

MIIE6TCCAtGgAwIBAgIJAP30nSyYUz/cMA0GCSqGSIb3DQEBCwUAMCYxJDAiBgNVBAMMG1Bvd2VyU2NhbGUgU0FNTCBTaWduaWSnIEtleTAeFw0yMjExMTUwMzU0NTFaFw0yMzExMTUwMzU0NTFaMCYxJDAiBgNVBAMMG1Bvd2VyU2NhbGUgU0FNTCBTaWduaWSnIEtleTCCAilwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMOOmYJ1aUuxvyH0nbUMurMbQubgtdpVBevy12D3qn+x7rgym8/v50da/4xpMmv/zbE0zJ0IVbWHZedibtQhLZ1qRSY/vBlaztU/nA90XQzXMnckzpcunOTG29SMO3x3Ud4*fqcP4sKhV

<snip>

Once regenerated, either the XML file or certificate can be downloaded, and the cluster configuration updated by either metadata download or manual copy:

Finally, upload the SP details back to the Identity provider.

For additional troubleshooting of OneFS SSO and authentication issues, there are a couple of key log files to check. These include:

Log file Information
/var/log/isi_saml_d.log SAML specific log messages logged by isi_saml_d.
/var/log/apache2/webui_httpd_error.log WebUI error messages including some SAML errors logged by the WebUI HTTP server.
/var/log/jwt.log Errors related to token generation logged by the JWT service.
/var/log/lsassd.log General authentication errors logged by the ‘lsassd’ service, such as failing to lookup users by email.

OneFS WebUI Single Sign-on Configuration and Deployment

In the first article in this series, we took a look at the architecture of the new OneFS WebUI SSO functionality. Now, we move on to its provisioning and setup.

SSO on PowerScale can be configured via either the OneFS WebUI or CLI. As such, OneFS 9.5 debuts a new dedicated WebUI SSO configuration page under Access > Authentication Providers > SSO. Alternatively, for command line afficionados, the CLI now includes a new ‘isi auth sso’ command set.

The overall SSO configuration flow is as above.

  1. Upgrade to OneFS 9.5

First, ensure the cluster is running OneFS 9.5 or a later release. If upgrading from an earlier OneFS version, note that the SSO service requires this upgrade to be committed prior to configuration and use.

Next, configure an SSO administrator. In OneFS, this account requires at least one of the following privileges:

Privilege Description
ISI_PRIV_LOGIN_PAPI Required for the admin to use the OneFS WebUI to administer SSO
ISI_PRIV_LOGIN_SSH Required for the admin to use the OneFS CLI via SSH to administer SSO.
ISI_PRIV_LOGIN_CONSOLE Required for the admin to use the OneFS CLI on the serial console to administer SSO.

The user account used for IdP management should have an associated email address configured.

  1. Setup Identity Provider (IdP)

OneFS SSO activation also requires having a suitable Identity Provider (IdP), such as ADFS, provisioned and available prior to setting up OneFS SSO.

ADFS can be configured via either the Windows GUI or command shell, and detailed information on the deployment and configuration of ADFS can be found in the Microsoft Windows Server documentation.

The Windows remote desktop utility (RDP) can be used to provision, connect to, and configure an ADFS server.

i.  When connected to ADFS, configure a rule defining access. For example, the following command line syntax can be used to create a simple rule that permits all users to log in:

$AuthRules = @"

@RuleTemplate="AllowAllAuthzRule" => issue(Type = "http://schemas.microsoft.com/

authorization/claims/permit", Value="true");

"@

Or from the ADFS UI:

Note that more complex rules can be crafted to meet the particular requirements of an organization, where desired.

ii.  Create a rule parameter to map the Active Directory (AD) user email address to the SAML NameID. For example:

$TransformRules = @"

@RuleTemplate = "LdapClaims"

@RuleName = "LDAP mail"

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/

windowsaccountname", Issuer == "AD AUTHORITY"]

=> issue(store = "Active Directory",

types = ("http://schemas.xmlsoap.org/ws/2005/05/identity/claims/ emailaddress"), query = ";mail;{0}", param = c.Value);

@RuleTemplate = "MapClaims"

@RuleName = "NameID"

c:[Type == "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress"]

=> issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/ nameidentifier", Issuer = c.

Issuer, OriginalIssuer = c.OriginalIssuer,

Value = c.Value, ValueType = c.ValueType, Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/ claimproperties/format"] = "urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress");

"@

iii.  Configure AD to trust the OneFS WebUI certificate.

iv.  Create the relying party trust. For example:

Add-AdfsRelyingPartyTrust -Name <cluster-name>\

-MetadataUrl "https://<cluster-node-ip>:8080/session/1/saml/metadata" \

-IssuanceAuthorizationRules $AuthRules -IssuanceTransformRules $TransformRules

Or from Windows Server Manager:

  1. Select Access Zone

Since OneFS SSO is zone-aware, the next step involves choosing the desired access zone to configure. Navigate to Access > Authentication providers > SSO, select an access zone (ie. the system zone) and click the ‘Add IdP’.

Note that each of a cluster’s access zone(s) must have an IdP configured for it. The same IdP can be used for all the zones, but each access zone must be configured separately.

  1. Add IdP Configuration

In OneFS 9.5 and later, the WebUI SSO configuration is a wizard-driven, ‘guided workflow’ process involving the following steps:

First, navigate to Access > Authentication providers > SSO, select the desired access zone (ie. the system zone), and then click the ‘Add IdP’ button.

On the ‘Add Identity Provider’ page, enter a unique name for the IdP. For example, Isln-IdP1 in this case:

When done, click ‘Next’, select the default ‘Upload metadata XML’ option, and browse to the XML file downloaded from the ADFS system:

Alternatively, if the preference is to enter the information by hand, select ‘Manual entry’ and complete the configuration form fields:

If the ‘manual’ method is selected, you must have the IdP certificate ready to upload. With the Manual option, the following information is required:

Field Description
Binding Select POST or Redirect binding.
Entity ID Unique identifier of the IdP as configured on the IdP. For example:

http://idp1.isilon.com/adfs/services/trust

Login URL Log in endpoint for the IdP. For example:

http://idp1.isilon.com/adfs/ls/

Logout URL Log out endpoint for the IdP. For example: http://idp1.example.com/adfs/ls/
Signing Certificate Provide the PEM encoded certificate obtained from the IdP. This certificate is required to verify messages from the IdP.

Upload the IdP certificate:

For example:

Repeat this step for each access zone in which SSO is to be configured.

When complete, click the ‘Next’ button to move on to the SP configuration step.

  1. Configure Service Provider

On the Service Provider page, confirm that the current access zone is carried over from the prior page.

Select ‘Metadata download’ or ‘Manual copy’, depending on the desired method of entering OneFS details about this SP to the IdP.

Provide the hostname or IP address for the SP for the current access zone.

Click the ‘Generate’ button to create the information (metadata) about OneFS and this access zone for to use in configuring the IdP.

This generated information can now be used to configure the IdP (in this case, Windows ADFS) to accept requests from PowerScale as the SP and its configured access zone.

As shown above, the ‘Edit Identity Provider’ WebUI page provides two methods:

Method Action
Metadata download Download the XML file which contains the signing certificate, etc.
Manual copy Use the ‘Copy Links’ in the lower half of the form to copy the information.

If ‘Manual copy’ is selected, the ‘Copy Links’ in the lower half of the form can be used to copy the information to the IdP.

Next, download the Signing Certificate.

When completed, click ‘Next’ to finish the SSO setup wizard.

  1. Enable SSO and Verify Operation

Once the IdP and SP are configured, a cluster admin can enable SSO per access zone via the OneFS WebUI by navigating to Access > Authentication providers > SSO. From here, select the desired access zone and click the ‘Enable SSO’ toggle:

Or from the OneFS CLI via the following syntax:

# isi auth sso settings modify --sso-enabled 1

OneFS WebUI Single Sign-on

The Security Assertion Markup Language (SAML) is an open standard for sharing security information about identity, authentication and authorization across different systems. SAML is implemented using the Extensible Markup Language (XML) standard for sharing data. The SAML framework enables Single Sign-on (SSO), which in turn allows users to log in once, and their login credential can be reused to authenticate with and access other different service providers. It defines several entities, described in the following table, including end users, service providers (SP), and identity providers (IdP), and is used to manage identity information.

Entity Description
End user An end user is an entity requiring authentication prior to being allowed to use an application.
Identity provider (IdP) Performs authentication and passes the user’s identity and authorization level to the service provider. For example, Active Directory Federation Services (ADFS).
Service provider (SP) Trusts the identity provider and authorizes the given user to access the requested resource.. With SAML 2.0, a PowerScale cluster is a service provider.
SAML Assertion XML document that the identity provider sends to the service provider that contains the user authorization.

The Windows Active Directory Federation Services (ADFS), for example, is one of the ubiquitous identity providers for SAML contexts.

OneFS 9.5 introduces SAML-based SSO for the WebUI to provide a more convenient authentication method, in addition to meeting the security compliance requirements for federal and enterprise customers. In OneFS 9.5, the WebUI’s initial login page has been redesigned to support SSO and, when enabled, a new ‘Log in with SSO’ button is displayed on the login page under the traditional username and password text boxes. For example:

OneFS SSO is also zone-aware in support of multi-tenant cluster configurations. As such, a separate Identity Provider can be configured independently for each OneFS access zone.

Under the hood, OneFS SSO employs the following high level architecture:

In OneFS 9.5, the SSO operates through HTTP REDIRECT and POST bindings, with the cluster acting as the Service Provider.

There are three different types of SAML Assertions – authentication, attribute, and authorization decision.

  • Authentication assertions prove identification of the user and provide the time the user logged in and what method of authentication they used (I.e., Kerberos, 2 factor, etc.)
  • The attribution assertion passes the SAML attributes to the service provider – SAML attributes are specific pieces of data that provide information about the user.
  • An authorization decision assertion says if the user is authorized to use the service or if the identify provider denied their request due to a password failure or lack of rights to the service.

SAML SSO works by transferring the user’s identity from one place (the identity provider) to another (the service provider). This is done through an exchange of digitally signed XML documents.

SAML Request, also known as an authentication request, is generated by the Service Provider to ‘request’ an authentication.

SAML Response is generated by the Identity Provider and contains the actual assertion of the authenticated user. In addition, a SAML Response may contain additional information, such as user profile information and group/role information, depending on what the Service Provider can support. Note that the Service Provider never directly interacts with the Identity Provider, with a browser acting as the agent facilitating any redirections.

Since SAML authentication is asynchronous, the Service Provider does not maintain the state of any authentication requests. As such, when the Service Provider receives a response from an Identity Provider, the response must contain all the necessary information.

The general flow is as follows:

When OneFS redirects a user to the configured IdP for login, it makes an HTTP GET request (SAMLRequest), instructing the IdP that the cluster is attempting to perform a login (SAMLAuthnRequest). When the user successfully authenticates, the IdP responds back to OneFS with an HTTP POST containing an HTML form (SAMLResponse) that indicates whether the login was successful, who logged in, plus any additional claims configured on the IdP.

On receiving the SAMLResponse, OneFS verifies the signature using the public key (X.509 certificate) in to ensure it really came from its trusted IdP and that none of the contents have been tampered with. OneFS then extracts the identity of the user, along with any other pertinent attributes. At this point, the user is redirected back to the OneFS WebUI dashboard (landing page), as if logged into the site manually.

In the next article in this series, we’ll take a detailed look at the above procedure to deploy SSO on a PowerScale cluster:

 

OneFS Account Security Policy

Another of the core security enhancements introduced in OneFS 9.5 is the ability to enforce strict user account security policies. This is required for compliance with both private and public sector security mandates. For example, the account policy restriction requirements expressed within the US military STIG requirements stipulate:

Requirement Description
Delay The OS must enforce a delay of at least 4 seconds between logon prompts following a failed logon attempt.
Disable The OS must disable account identifiers (individuals, groups, roles, and devices) after 35 days of inactivity.
Limit The OS must limit the number of concurrent sessions to ten for all accounts and/or account types.

To directly address these security edicts, OneFS 9.5 adds the following account policy restriction controls:

Account Policy Function Details
Delay after failed login • After a failed login, OneFS enforces a configurable delay for subsequent logins on same cluster node

• Only applicable to administrative logins (not Protocol logins)

 

Disable inactive accounts • Disables an inactive account after specified number of days.

• Only applicable to Local user accounts

• Cluster wide

Concurrent session limit • Limits the number of active sessions a user can have on a cluster node

• Only applicable to administrative logins

• Node specific

OneFS provides a variety of access mechanisms for administering a cluster. These include SSH, serial console, WebUI and platform API, all of which utilize different underlying access methods. The serial console and SSH are standard FreeBSD third-party applications and are accounted for per-node, whereas the WebUI and pAPI use HTTP module extensions to facilitate access to the system and services and are accounted for cluster-wide. Prior to OneFS 9.5, there was no common mechanism to represent or account for sessions across these disparate applications.

Under the hood, the OneFS account security policy framework encompasses the following high-level architecture:

With SSH, there’s no explicit or reliable ‘log-off’ event sent to OneFS, beyond actually disconnecting the connection. As such, accounting for active sessions can be problematic and unreliable, especially when connections time out or unexpectedly disconnect. However, OneFS does include an accounting database that stores records of system activities like user login and logout, which can be queried to determine active SSH sessions. Each active SSH connection has an isi_ssh_d process owned by the account associated with it, and this information can be gathered via standard syscalls. OneFS enumerates the number of SSHD processes per account to calculate the total number of active established sessions. This value is then used as part of the total concurrent administrative sessions limit. Since SSH only supports user access via the system zone, there is no need for any zone-aware accounting.

The WebUI and platform API use JSON web tokens (JWT) for authenticated sessions. OneFS stores the JWT tokens in the cluster-wide kvstore, and access policy uses valid session tokens in the kvstore to account for active sessions when user logs on via the WebUI or pAPI. When the user logs off, the associated token is removed, and a message is sent to JWT service with an explicit log off notification. In the event that a session times out or disconnects, the JWT service will not get an event, but the tokens have a limit short lifespan and any expired tokens are purged from the list on a scheduled basis in conjunction with the JWT timer. OneFS enumerates the unique session IDs associated with each user’s JWT tokens in the kvstore to get a number of active WebUI and pAPI sessions to use as part of user’s session limit check.

For serial console access accounting, the process table will have information when an STTY connection is active and OneFS extrapolates user data from it to determine the session count, similar to ssh with a syscall for process data. There is an accounting database that stores records of system activities like user login and logout, which is also queried for active console sessions. Serial console access is only from the system zone, so there is no need for zone-aware accounting.

An API call retrieves user session data from the process table and kvstore to calculate number of user active sessions. As such, the checking and enforcement of session limits is performed in similar manner to the verification of user privileges for SSH, serial console, or WebUI access.

Delaying Failed Login Reconnections

OneFS 9.5 provides the ability to enforce a configurable delay period. This delay is specified in seconds, after which every unsuccessful authentication attempt results in the user being denied the ability to reconnect to the cluster until after the configured delay period has passed. The login delay period is defined in seconds through the ‘FailedLoginDelayTime’ global attribute and, by default, OneFS is configured for no delay via a ‘FailedLoginDelayTime’ value of ‘0’. When a cluster is placed into hardened mode with the STIG policy enacted, the delay value is automatically set to 4 seconds. Note that the delay happens in the lsass client, so that the authentication service is not affected.

The configured ‘failed login delay time’ limit can be viewed with following CLI command:

# isi auth settings global view

                            Send NTLMv2: No

                      Space Replacement:

                              Workgroup: WORKGROUP

               Provider Hostname Lookup: disabled

                          Alloc Retries: 5

                 User Object Cache Size: 47.68M

                       On Disk Identity: native

                         RPC Block Time: Now

                       RPC Max Requests: 64

                            RPC Timeout: 30s

Default LDAP TLS Revocation Check Level: none

                   System GID Threshold: 80

                   System UID Threshold: 80

                         Min Mapped Rid: 2147483648

                              Group UID: 4294967292

                               Null GID: 4294967293

                               Null UID: 4294967293

                            Unknown GID: 4294967294

                            Unknown UID: 4294967294

                Failed Login Delay Time: Now

               Concurrent Session Limit: 0

Similarly, the following syntax will configure the ‘failed login delay time’ to a value of 4 seconds:

# isi auth settings global modify --failed-login-delay-time 4s

# isi auth settings global view | grep -i delay

                Failed Login Delay Time: 4s

However, when a cluster is put into STIG hardening mode, the ‘Concurrent Session Limit’ is automatically set to ’10’.

# isi auth settings global view | grep -i delay

                Failed Login Delay Time: 10s

The delay time after login failure can also be configured from the WebUI under Access > Settings > Global provider settings:

The valid range of the ‘FailedLoginDelayTime’ global attribute is from ‘0’ to ‘65535’, and the delay time is limited to the same cluster node.

Note that this maximum session limit is only applicable to administrative logins.

Disabling Inactive Accounts

In OneFS 9.5, if desired, any user account that has been inactive for a configurable duration will be automatically disabled. Once disabled, administrative intervention is required in order to re-enable a deactivated user account. The last activity time of a user is determined by their previous logon, and a timer runs every midnight during which all ‘inactive’ accounts are disabled. If the last logon record for a user is unavailable, or stale, the timestamp when the account was enabled is taken as their last activity instead. If inactivity tracking is enabled after the last logon (or enabled) time of a user, the inactivity tracking time is considered for inactivity period.
This feature is disabled by default in OneFS, and all users are exempted from inactivity tracking until configured otherwise. However, individual accounts can be exempted from this behavior, and this can be configured through the user-specific ‘DisableWhenInactive’ attribute. For example:

# isi auth user view user1 | grep -i inactive

   Disable When Inactive: Yes

# isi auth user modify user1 --disable-when-inactive 0

# isi auth user view user1 | grep -i inactive

   Disable When Inactive: No

If a cluster is put into STIG hardened mode, the value for the ‘MaxInactivityDays’ parameter is automatically reconfigured to value ‘35’, implying a user will be disabled after 35 days of inactivity. All the local users are removed from exemption when in STIG hardened mode.

Note that this functionality is limited to only the ’local’ provider and does not apply to ‘file’ providers.

The inactive account disabling configuration can be viewed from the CLI with the following syntax. In this example, the ‘MaxInactivityDays’ attribute is configured for 35 days:

# isi auth local view system

                    Name: System

                  Status: active

          Authentication: Yes

   Create Home Directory: Yes

 Home Directory Template: /ifs/home/%U

        Lockout Duration: Now

       Lockout Threshold: 0

          Lockout Window: Now

             Login Shell: /bin/zsh

            Machine Name:

        Min Password Age: Now

        Max Password Age: 4W

     Min Password Length: 15

    Password Prompt Time: 2W

     Password Complexity: -

 Password History Length: 0

  Password Chars Changed: 8

Password Percent Changed: 50

      Password Hash Type: NTHash

     Max Inactivity Days: 35

Inactive account disabling can also be configured from the WebUI under Access > Authentication providers > Local provider:

The valid range of the ‘MaxInactivityDays’ parameter is from ‘0’ to ‘UINT_MAX’. As such, the following CLI syntax will configure the maximum number of days a user account can be inactive before it will be disabled to 10 days:

# isi auth local modify system --max-inactivity-days 10

# isi auth local view system | grep -i inactiv

     Max Inactivity Days: 0tem –max-inactivity-days 10

Setting this value to 0 days will disable the feature:

# isi auth local modify system --max-inactivity-days 0

# isi auth local view system | grep -i inactiv

     Max Inactivity Days: 0tem –max-inactivity-days 0

Inactivity account disabling, as well as password expiry, can also be configured granularly, per user account. For example, ‘user1’ has a default configuration of the ‘disable when inactive’ threshold set to ‘No’.

# isi auth users view user1

                    Name: user1

                      DN: CN=user1,CN=Users,DC=GLADOS

              DNS Domain: -

                  Domain: GLADOS

                Provider: lsa-local-provider:System

        Sam Account Name: user1

                     UID: 2000

                     SID: S-1-5-21-1839173366-2940572996-2365153926-1000

                 Enabled: Yes

                 Expired: No

                  Expiry: -

                  Locked: No

                   Email: -

                   GECOS: -

           Generated GID: No

           Generated UID: No

           Generated UPN: Yes

           Primary Group

                          ID: GID:1800

                        Name: Isilon Users

          Home Directory: /ifs/home/user1

        Max Password Age: 4W

        Password Expired: No

         Password Expiry: 2023-06-15T17:45:55

       Password Last Set: 2023-05-18T17:45:55

        Password Expired: No

              Last Logon: -

                   Shell: /bin/zsh

                     UPN: user1@GLADOS

User Can Change Password: Yes

   Disable When Inactive: No

The following CLI command will activate the account inactivity disabling setting and enable password expiry for the ‘user1’ account:

# isi auth users modify user1 --disable-when-inactive Yes --password-expires Yes

Inactive account disabling can also be configured from the WebUI under Access > Membership and roles > Users > Providers:

Limiting Concurrent Sessions

OneFS 9.5 can limit the number of administrative sessions active on a OneFS cluster node, and all WebUI, SSH, pAPI, and serial console sessions are accounted for when calculating the session limit. The SSH and console session count is node-local, whereas WebUI and pAPI sessions are tracked cluster-wide. As such, the formula used to calculate a node’s total active sessions is as follows:

Total active user sessions on a node = Total WebUI and pAPI sessions across the cluster + Total SSH and Console sessions on the node

This feature leverages the cluster wide session management through JWT for calculating the total number of sessions on a cluster’s node. By default, OneFS 9.5 has is no configured limit and the ‘Concurrent Session Limit’ parameter has a value of ‘0’. For example:

# isi auth settings global view

                            Send NTLMv2: No

                      Space Replacement:

                              Workgroup: WORKGROUP

               Provider Hostname Lookup: disabled

                          Alloc Retries: 5

                 User Object Cache Size: 47.68M

                       On Disk Identity: native

                         RPC Block Time: Now

                       RPC Max Requests: 64

                            RPC Timeout: 30s

Default LDAP TLS Revocation Check Level: none

                   System GID Threshold: 80

                   System UID Threshold: 80

                         Min Mapped Rid: 2147483648

                              Group UID: 4294967292

                               Null GID: 4294967293

                               Null UID: 4294967293

                            Unknown GID: 4294967294

                            Unknown UID: 4294967294

                Failed Login Delay Time: Now

               Concurrent Session Limit: 0

The following CLI syntax will configure the ‘concurrent session limit’ to a value of 5 sessions:

# isi auth settings global modify –-concurrent-session-limit 5

# isi auth settings global view | grep -i concur

                Concurrent Session Limit: 5

Once the session limit has been exceeded, attempts to connect, in this case as ‘root’ via SSH, will be met with the following ‘Access denied’ error message:

login as: root

Keyboard-interactive authentication prompts from server:

| Password:

End of keyboard-interactive prompts from server

Access denied

password:

The concurrent sessions limit can also be configured from the WebUI under Access > Settings > Global provider settings:

However, when a cluster is put into STIG hardening mode, the ‘Concurrent Session Limit’ is automatically set to a maximum of 10 sessions.

Note that this maximum session limit is only applicable to administrative logins.

Performance

Disabling an account after a period of inactivity in OneFS requires a SQLite database update every time a user has successfully logged on to the OneFS cluster. After a successful logon, the time to logon is recorded in the database, which is later used to compute the inactivity period.

Inactivity tracking is disabled by default in OneFS 9.5, but can be easily enabled by configuring the ‘MaxInactivityDays’ attribute to a non-zero value. In cases where inactivity tracking is enabled and many users are not exempt from inactivity tracking, a significant number of logons within a short period of time can generate significant SQLite database requests. However, OneFS consolidates multiple database updates during user logon to a single commit in order to minimize the overall load.

Troubleshooting

When it comes to troubleshooting OneFS account security policy configurations there are two main logfiles to check:

  • /var/log/lsassd.log
  • /var/log/messages
  • /var/log/isi_papi_d.log

For additional reporting detail, debug level logging can be enabled on the lsassd.log file with the following CLI command:

# /usr/likewise/bin/lwsm set-log-level lsass - debug

When finished, logging can be returned to the regular ‘error’ level:

# /usr/likewise/bin/lwsm set-log-level lsass - error

OneFS Password Security Policy

Among the slew of security enhancements introduced in OneFS 9.5 is the ability to mandate a more stringent password policy. This is required in order to comply with security requirements such as the US military STIG, which stipulates:

Requirement Description
Length An OS or network device must enforce a minimum 15-character password length.
Percentage An OS must require the change of at least 50% of the total number of characters when passwords are changed.
Position A network device must require that when a password is changed, the characters are changed in at least eight of the positions within the password.
Temporary password The OS must allow the use of a temporary password for system logons with an immediate change to a permanent password.

The OneFS password security architecture can be summarized as follows:

Within the OneFS security subsystem, authentication is handled in OneFS by LSASSD, the daemon used to service authentication requests for lwiod.

Component Description
LSASSD The local security authority subsystem service (LSASS) handles authentication and identity management as users connect to the cluster.
File provider The file provider includes users from /etc/password and groups from /etc/groups.
Local provider The local provider includes local cluster accounts like ‘anonymous’, ‘guest’, etc.
SSHD OpenSSH Daemon which provides secure encrypted communications between a client and a cluster node over an insecure network.
pAPI The OneFS Platform API (PAPI), which provides programmatic interfaces to OneFS configuration and management via a RESTful HTTPS service.

In OneFS AIMA, there are several different kinds of backend providers: Local provider, file provider, AD provider, NIS provider, etc. Each provider is responsible for the management of users and groups inside the provider. For OneFS password policy enforcement, the Local and File providers are the focus.

The local provider is based on an SamDB style file stored with prefix path of “/ifs/.ifsvar”, and its provider settings can be viewed by the following CLI syntax:

# isi auth local view System

On the other hand, the file provider is based on the FreeBSD spwd.db file, and its configuration can be viewed by the following CLI command:

# isi auth file view System

Each provider stores and manage its own users. For the local provider, ` isi auth users create’ CLI command will create a user inside the provider by default. However, for the file provider, there is no corresponding command. Instead,  the `pw` CLI command can be used to create a new file provider user.

After the user is created, the `isi auth users modify <USER>` CLI command can be used to change the attributes of the user for both the file and local providers. However, not all attributes are supported for both providers. For example, the file provider does not support password expiry.

 

The fundamental password policy CLI changes introduced in OneFS 9.5 are as follows:

Operation OneFS 9.5 Change Details
change-password Modified Needed to provide old password for changing so that we can calculate how many chars/percent changed
reset-password Added Generates a temp password that meets current password policy for user to login
set-password Deprecated Doesn’t need to provide old password

A user’s password can now be set, changed, and reset by either ‘root’ or ‘admin’. This is supported by the new ‘isi auth users change-password’ or ‘isi auth users reset-password’ CLI command syntax. The latter, for example, returns a temporary password and requires the user to change it on next login. After logging in with the temporary (albeit secure) password, OneFS immediately forces the user to change it:

# whoami
admin

# isi auth users reset-password user1
4$_x\d\Q6V9E:sH

# ssh user1@localhost
(user1@localhost) Password:
(user1@localhost) Your password has expired.
You are required to immediately change your password.
Changing password for user1
New password:
(user1@localhost) Re-enter password:
Last login: Wed May 17 08:02:47 from 127.0.0.1
PowerScale OneFS 9.5.0.0

# whoami
user1

Also in OneFS 9.5 and later, the CLI ‘isi auth local view system’ command sees the addition of four new fields:

  • Password Chars Changed
  • Password Percent Changed
  • Password Hash Type
  • Max Inactivity Days

For example:

# isi auth local view system
                    Name: System
                  Status: active
          Authentication: Yes
   Create Home Directory: Yes
 Home Directory Template: /ifs/home/%U
        Lockout Duration: Now
       Lockout Threshold: 0
          Lockout Window: Now
             Login Shell: /bin/zsh
            Machine Name:
        Min Password Age: Now
        Max Password Age: 4W
     Min Password Length: 0
    Password Prompt Time: 2W
     Password Complexity: -
 Password History Length: 0
  Password Chars Changed: 0
Password Percent Changed: 0
      Password Hash Type: NTHash
     Max Inactivity Days: 0

The following CLI command syntax configures OneFS to require a minimum password length of 15 characters, a 50% or greater change, and 8 or more characters to be altered for a successful password reset:

# isi auth local modify system --min-password-length 15 --password-chars-changed 8 --password-percent-changed 50

Next, a command is issued to create a new user, ‘user2’, with a 10 character password:

# isi auth users create user2 --password 0123456789

Failed to add user user1: The specified password does not meet the configured password complexity or history requirements

This attempt fails because the password does not meet the configured password criteria (15 chars, 50% change, 8 chars to be altered).

Instead, the password for the new account, ‘user2’, is set to an appropriate value: ie  ‘0123456789abcdef’. Also, the ‘–prompt-password-change’ flag is included to force the user to change their password on next login.

# isi auth users create user2 --password 0123456789abcdef –prompt-password-change 1

On logging in to the ‘user2’ account, OneFS immediately prompts for a new password. In the example below, the following non-compliant password (‘012345678zyxw’) is entered.

  • 0123456789abcdef -> 012345678zyxw = Failure

This returns an unsuccessful change attempt failure, since it does not meet the 15 character minimum:

# su user2
New password:
Re-enter password:
The specified password does not meet the configured password complexity requirements.
Your password must meet the following requirements:
  * Must contain at least 15 characters.
  * Must change at least 8 characters.
  * Must change at least 50% of characters.
New password:

Instead, a compliant password and successful change could be:

  • 0123456789abcdef -> 0123456zyxwvuts = Success

The following command can also be used to change the password for a user. For example, to update user2’s password:

# isi auth users change-password user2
Current password (hit enter if none):
New password:
Confirm new password:

If a non-compliant password is entered, the following error is returned:

Password change failed: The specified password does not meet the configured password complexity or history requirements

When employed, OneFS hardening automatically enforces security-based configurations. The hardening engine is profile-based, and its STIG security profile is predicated on security mandates specified in the US Department of Defense (DoD) Security Requirements Guides (SRGs) and Security Technical Implementation Guides (STIGs).

On applying the STIG hardening security profile to a cluster (‘isi hardening apply –profile=STIG’), the password policy settings are automatically reconfigured to the following values:

Field Normal Value STIG Hardened
Lockout Duration Now Now
Lockout Threshold 0 3
Lockout Window Now 15m
Min Password Age Now 1D
Max Password Age 4W 8W4D
Min Password Length 0 15
Password Prompt Time 2W 2W
Password Complexity lowercase, numeric, repeat, symbol, uppercase
Password History Length 0 5
Password Chars Changed 0 8
Password Percent Changed 0 50
Password Hash Type NTHash SHA512
Max Inactivity Days 0 35

For example:

# uname -or
Isilon OneFS 9.5.0.0

# isi hardening list
Name  Description                       Status
---------------------------------------------------
STIG  Enable all STIG security settings Applied
---------------------------------------------------
Total: 1

# isi auth local view system
                    Name: System
                  Status: active
          Authentication: Yes
   Create Home Directory: Yes
 Home Directory Template: /ifs/home/%U
        Lockout Duration: Now
       Lockout Threshold: 3
          Lockout Window: 15m
             Login Shell: /bin/zsh
            Machine Name:
        Min Password Age: 1D
        Max Password Age: 8W4D
     Min Password Length: 15
    Password Prompt Time: 2W
     Password Complexity: lowercase, numeric, repeat, symbol, uppercase
 Password History Length: 5
  Password Chars Changed: 8
Password Percent Changed: 50
      Password Hash Type: SHA512
     Max Inactivity Days: 35

Note that the ‘Password Hash Type’ is changed from the default ‘NTHash’ to the more secure ‘SHA512’ encoding, in addition to setting the various password criteria.

The OneFS 9.5 WebUI also sees several additions and alterations to the Password Policy page. These include:

Operation OneFS 9.5 Change Details
Policy page Added New Password policy page under Access -> Membership and Roles
reset-password Added Generate a random password that meets current password policy for user to login

The most obvious change is the transfer of the policy configuration elements from the local provider page to a new dedicated Password Policy page.

Here’s the OneFS 9.4 ‘View a local provider’ page, under Access > Authentication providers > Local providers > System:

The above is replaced and augmented in the OneFS 9.5 WebUI with the following page, located under Access > Membership and Roles > Password Policy:

New password policy configuration options are included to require upper-case, lower-case, numeric, or special characters and limit the number of contiguous repeats of a character, etc.

When it comes to changing a password, only a permitted user can make their change. This can be performed from a couple of locations in the WebUI. First, the user options on the task bar at the top of each screen now provides a ‘Change password’ option:

A pop-up warning message will also be displayed by the WebUI, informing when password expiration is imminent. This warning provides a ‘Change Password’ link:

Clicking on the above link displays the following page:

A new password complexity tool-tip message is also displayed, informing the user of safe password selection.

Note that re-login is required after a password change.

On the ‘Users’ page under Access > Membership and roles > Users, the ‘Action’ drop-down list on the ‘Users’ page now also contains a ‘Reset Password’ option:

The successful reset confirmation pop-up offers both a ‘show’ and ‘copy’ option, while informing the cluster administrator to share the new password with the user, and for them to change their password during their next login:

The ‘Create user’ page now provides an additional field that requires password confirmation. Additionally, the password complexity tool-tip message is also displayed:

The redesigned ‘Edit user details’ page no longer provides a field to edit the password directly:

Instead, the ‘Action’ drop-down list on the ‘Users’ page now contains a ‘Reset Password’ option.