OneFS Account Security Policy

Another of the core security enhancements introduced in OneFS 9.5 is the ability to enforce strict user account security policies. This is required for compliance with both private and public sector security mandates. For example, the account policy restriction requirements expressed within the US military STIG requirements stipulate:

Requirement Description
Delay The OS must enforce a delay of at least 4 seconds between logon prompts following a failed logon attempt.
Disable The OS must disable account identifiers (individuals, groups, roles, and devices) after 35 days of inactivity.
Limit The OS must limit the number of concurrent sessions to ten for all accounts and/or account types.

To directly address these security edicts, OneFS 9.5 adds the following account policy restriction controls:

Account Policy Function Details
Delay after failed login • After a failed login, OneFS enforces a configurable delay for subsequent logins on same cluster node

• Only applicable to administrative logins (not Protocol logins)

 

Disable inactive accounts • Disables an inactive account after specified number of days.

• Only applicable to Local user accounts

• Cluster wide

Concurrent session limit • Limits the number of active sessions a user can have on a cluster node

• Only applicable to administrative logins

• Node specific

OneFS provides a variety of access mechanisms for administering a cluster. These include SSH, serial console, WebUI and platform API, all of which utilize different underlying access methods. The serial console and SSH are standard FreeBSD third-party applications and are accounted for per-node, whereas the WebUI and pAPI use HTTP module extensions to facilitate access to the system and services and are accounted for cluster-wide. Prior to OneFS 9.5, there was no common mechanism to represent or account for sessions across these disparate applications.

Under the hood, the OneFS account security policy framework encompasses the following high-level architecture:

With SSH, there’s no explicit or reliable ‘log-off’ event sent to OneFS, beyond actually disconnecting the connection. As such, accounting for active sessions can be problematic and unreliable, especially when connections time out or unexpectedly disconnect. However, OneFS does include an accounting database that stores records of system activities like user login and logout, which can be queried to determine active SSH sessions. Each active SSH connection has an isi_ssh_d process owned by the account associated with it, and this information can be gathered via standard syscalls. OneFS enumerates the number of SSHD processes per account to calculate the total number of active established sessions. This value is then used as part of the total concurrent administrative sessions limit. Since SSH only supports user access via the system zone, there is no need for any zone-aware accounting.

The WebUI and platform API use JSON web tokens (JWT) for authenticated sessions. OneFS stores the JWT tokens in the cluster-wide kvstore, and access policy uses valid session tokens in the kvstore to account for active sessions when user logs on via the WebUI or pAPI. When the user logs off, the associated token is removed, and a message is sent to JWT service with an explicit log off notification. In the event that a session times out or disconnects, the JWT service will not get an event, but the tokens have a limit short lifespan and any expired tokens are purged from the list on a scheduled basis in conjunction with the JWT timer. OneFS enumerates the unique session IDs associated with each user’s JWT tokens in the kvstore to get a number of active WebUI and pAPI sessions to use as part of user’s session limit check.

For serial console access accounting, the process table will have information when an STTY connection is active and OneFS extrapolates user data from it to determine the session count, similar to ssh with a syscall for process data. There is an accounting database that stores records of system activities like user login and logout, which is also queried for active console sessions. Serial console access is only from the system zone, so there is no need for zone-aware accounting.

An API call retrieves user session data from the process table and kvstore to calculate number of user active sessions. As such, the checking and enforcement of session limits is performed in similar manner to the verification of user privileges for SSH, serial console, or WebUI access.

Delaying Failed Login Reconnections

OneFS 9.5 provides the ability to enforce a configurable delay period. This delay is specified in seconds, after which every unsuccessful authentication attempt results in the user being denied the ability to reconnect to the cluster until after the configured delay period has passed. The login delay period is defined in seconds through the ‘FailedLoginDelayTime’ global attribute and, by default, OneFS is configured for no delay via a ‘FailedLoginDelayTime’ value of ‘0’. When a cluster is placed into hardened mode with the STIG policy enacted, the delay value is automatically set to 4 seconds. Note that the delay happens in the lsass client, so that the authentication service is not affected.

The configured ‘failed login delay time’ limit can be viewed with following CLI command:

# isi auth settings global view

                            Send NTLMv2: No

                      Space Replacement:

                              Workgroup: WORKGROUP

               Provider Hostname Lookup: disabled

                          Alloc Retries: 5

                 User Object Cache Size: 47.68M

                       On Disk Identity: native

                         RPC Block Time: Now

                       RPC Max Requests: 64

                            RPC Timeout: 30s

Default LDAP TLS Revocation Check Level: none

                   System GID Threshold: 80

                   System UID Threshold: 80

                         Min Mapped Rid: 2147483648

                              Group UID: 4294967292

                               Null GID: 4294967293

                               Null UID: 4294967293

                            Unknown GID: 4294967294

                            Unknown UID: 4294967294

                Failed Login Delay Time: Now

               Concurrent Session Limit: 0

Similarly, the following syntax will configure the ‘failed login delay time’ to a value of 4 seconds:

# isi auth settings global modify --failed-login-delay-time 4s

# isi auth settings global view | grep -i delay

                Failed Login Delay Time: 4s

However, when a cluster is put into STIG hardening mode, the ‘Concurrent Session Limit’ is automatically set to ’10’.

# isi auth settings global view | grep -i delay

                Failed Login Delay Time: 10s

The delay time after login failure can also be configured from the WebUI under Access > Settings > Global provider settings:

The valid range of the ‘FailedLoginDelayTime’ global attribute is from ‘0’ to ‘65535’, and the delay time is limited to the same cluster node.

Note that this maximum session limit is only applicable to administrative logins.

Disabling Inactive Accounts

In OneFS 9.5, if desired, any user account that has been inactive for a configurable duration will be automatically disabled. Once disabled, administrative intervention is required in order to re-enable a deactivated user account. The last activity time of a user is determined by their previous logon, and a timer runs every midnight during which all ‘inactive’ accounts are disabled. If the last logon record for a user is unavailable, or stale, the timestamp when the account was enabled is taken as their last activity instead. If inactivity tracking is enabled after the last logon (or enabled) time of a user, the inactivity tracking time is considered for inactivity period.
This feature is disabled by default in OneFS, and all users are exempted from inactivity tracking until configured otherwise. However, individual accounts can be exempted from this behavior, and this can be configured through the user-specific ‘DisableWhenInactive’ attribute. For example:

# isi auth user view user1 | grep -i inactive

   Disable When Inactive: Yes

# isi auth user modify user1 --disable-when-inactive 0

# isi auth user view user1 | grep -i inactive

   Disable When Inactive: No

If a cluster is put into STIG hardened mode, the value for the ‘MaxInactivityDays’ parameter is automatically reconfigured to value ‘35’, implying a user will be disabled after 35 days of inactivity. All the local users are removed from exemption when in STIG hardened mode.

Note that this functionality is limited to only the ’local’ provider and does not apply to ‘file’ providers.

The inactive account disabling configuration can be viewed from the CLI with the following syntax. In this example, the ‘MaxInactivityDays’ attribute is configured for 35 days:

# isi auth local view system

                    Name: System

                  Status: active

          Authentication: Yes

   Create Home Directory: Yes

 Home Directory Template: /ifs/home/%U

        Lockout Duration: Now

       Lockout Threshold: 0

          Lockout Window: Now

             Login Shell: /bin/zsh

            Machine Name:

        Min Password Age: Now

        Max Password Age: 4W

     Min Password Length: 15

    Password Prompt Time: 2W

     Password Complexity: -

 Password History Length: 0

  Password Chars Changed: 8

Password Percent Changed: 50

      Password Hash Type: NTHash

     Max Inactivity Days: 35

Inactive account disabling can also be configured from the WebUI under Access > Authentication providers > Local provider:

The valid range of the ‘MaxInactivityDays’ parameter is from ‘0’ to ‘UINT_MAX’. As such, the following CLI syntax will configure the maximum number of days a user account can be inactive before it will be disabled to 10 days:

# isi auth local modify system --max-inactivity-days 10

# isi auth local view system | grep -i inactiv

     Max Inactivity Days: 0tem –max-inactivity-days 10

Setting this value to 0 days will disable the feature:

# isi auth local modify system --max-inactivity-days 0

# isi auth local view system | grep -i inactiv

     Max Inactivity Days: 0tem –max-inactivity-days 0

Inactivity account disabling, as well as password expiry, can also be configured granularly, per user account. For example, ‘user1’ has a default configuration of the ‘disable when inactive’ threshold set to ‘No’.

# isi auth users view user1

                    Name: user1

                      DN: CN=user1,CN=Users,DC=GLADOS

              DNS Domain: -

                  Domain: GLADOS

                Provider: lsa-local-provider:System

        Sam Account Name: user1

                     UID: 2000

                     SID: S-1-5-21-1839173366-2940572996-2365153926-1000

                 Enabled: Yes

                 Expired: No

                  Expiry: -

                  Locked: No

                   Email: -

                   GECOS: -

           Generated GID: No

           Generated UID: No

           Generated UPN: Yes

           Primary Group

                          ID: GID:1800

                        Name: Isilon Users

          Home Directory: /ifs/home/user1

        Max Password Age: 4W

        Password Expired: No

         Password Expiry: 2023-06-15T17:45:55

       Password Last Set: 2023-05-18T17:45:55

        Password Expired: No

              Last Logon: -

                   Shell: /bin/zsh

                     UPN: user1@GLADOS

User Can Change Password: Yes

   Disable When Inactive: No

The following CLI command will activate the account inactivity disabling setting and enable password expiry for the ‘user1’ account:

# isi auth users modify user1 --disable-when-inactive Yes --password-expires Yes

Inactive account disabling can also be configured from the WebUI under Access > Membership and roles > Users > Providers:

Limiting Concurrent Sessions

OneFS 9.5 can limit the number of administrative sessions active on a OneFS cluster node, and all WebUI, SSH, pAPI, and serial console sessions are accounted for when calculating the session limit. The SSH and console session count is node-local, whereas WebUI and pAPI sessions are tracked cluster-wide. As such, the formula used to calculate a node’s total active sessions is as follows:

Total active user sessions on a node = Total WebUI and pAPI sessions across the cluster + Total SSH and Console sessions on the node

This feature leverages the cluster wide session management through JWT for calculating the total number of sessions on a cluster’s node. By default, OneFS 9.5 has is no configured limit and the ‘Concurrent Session Limit’ parameter has a value of ‘0’. For example:

# isi auth settings global view

                            Send NTLMv2: No

                      Space Replacement:

                              Workgroup: WORKGROUP

               Provider Hostname Lookup: disabled

                          Alloc Retries: 5

                 User Object Cache Size: 47.68M

                       On Disk Identity: native

                         RPC Block Time: Now

                       RPC Max Requests: 64

                            RPC Timeout: 30s

Default LDAP TLS Revocation Check Level: none

                   System GID Threshold: 80

                   System UID Threshold: 80

                         Min Mapped Rid: 2147483648

                              Group UID: 4294967292

                               Null GID: 4294967293

                               Null UID: 4294967293

                            Unknown GID: 4294967294

                            Unknown UID: 4294967294

                Failed Login Delay Time: Now

               Concurrent Session Limit: 0

The following CLI syntax will configure the ‘concurrent session limit’ to a value of 5 sessions:

# isi auth settings global modify –-concurrent-session-limit 5

# isi auth settings global view | grep -i concur

                Concurrent Session Limit: 5

Once the session limit has been exceeded, attempts to connect, in this case as ‘root’ via SSH, will be met with the following ‘Access denied’ error message:

login as: root

Keyboard-interactive authentication prompts from server:

| Password:

End of keyboard-interactive prompts from server

Access denied

password:

The concurrent sessions limit can also be configured from the WebUI under Access > Settings > Global provider settings:

However, when a cluster is put into STIG hardening mode, the ‘Concurrent Session Limit’ is automatically set to a maximum of 10 sessions.

Note that this maximum session limit is only applicable to administrative logins.

Performance

Disabling an account after a period of inactivity in OneFS requires a SQLite database update every time a user has successfully logged on to the OneFS cluster. After a successful logon, the time to logon is recorded in the database, which is later used to compute the inactivity period.

Inactivity tracking is disabled by default in OneFS 9.5, but can be easily enabled by configuring the ‘MaxInactivityDays’ attribute to a non-zero value. In cases where inactivity tracking is enabled and many users are not exempt from inactivity tracking, a significant number of logons within a short period of time can generate significant SQLite database requests. However, OneFS consolidates multiple database updates during user logon to a single commit in order to minimize the overall load.

Troubleshooting

When it comes to troubleshooting OneFS account security policy configurations there are two main logfiles to check:

  • /var/log/lsassd.log
  • /var/log/messages
  • /var/log/isi_papi_d.log

For additional reporting detail, debug level logging can be enabled on the lsassd.log file with the following CLI command:

# /usr/likewise/bin/lwsm set-log-level lsass - debug

When finished, logging can be returned to the regular ‘error’ level:

# /usr/likewise/bin/lwsm set-log-level lsass - error

OneFS Password Security Policy

Among the slew of security enhancements introduced in OneFS 9.5 is the ability to mandate a more stringent password policy. This is required in order to comply with security requirements such as the US military STIG, which stipulates:

Requirement Description
Length An OS or network device must enforce a minimum 15-character password length.
Percentage An OS must require the change of at least 50% of the total number of characters when passwords are changed.
Position A network device must require that when a password is changed, the characters are changed in at least eight of the positions within the password.
Temporary password The OS must allow the use of a temporary password for system logons with an immediate change to a permanent password.

The OneFS password security architecture can be summarized as follows:

Within the OneFS security subsystem, authentication is handled in OneFS by LSASSD, the daemon used to service authentication requests for lwiod.

Component Description
LSASSD The local security authority subsystem service (LSASS) handles authentication and identity management as users connect to the cluster.
File provider The file provider includes users from /etc/password and groups from /etc/groups.
Local provider The local provider includes local cluster accounts like ‘anonymous’, ‘guest’, etc.
SSHD OpenSSH Daemon which provides secure encrypted communications between a client and a cluster node over an insecure network.
pAPI The OneFS Platform API (PAPI), which provides programmatic interfaces to OneFS configuration and management via a RESTful HTTPS service.

In OneFS AIMA, there are several different kinds of backend providers: Local provider, file provider, AD provider, NIS provider, etc. Each provider is responsible for the management of users and groups inside the provider. For OneFS password policy enforcement, the Local and File providers are the focus.

The local provider is based on an SamDB style file stored with prefix path of “/ifs/.ifsvar”, and its provider settings can be viewed by the following CLI syntax:

# isi auth local view System

On the other hand, the file provider is based on the FreeBSD spwd.db file, and its configuration can be viewed by the following CLI command:

# isi auth file view System

Each provider stores and manage its own users. For the local provider, ` isi auth users create’ CLI command will create a user inside the provider by default. However, for the file provider, there is no corresponding command. Instead,  the `pw` CLI command can be used to create a new file provider user.

After the user is created, the `isi auth users modify <USER>` CLI command can be used to change the attributes of the user for both the file and local providers. However, not all attributes are supported for both providers. For example, the file provider does not support password expiry.

 

The fundamental password policy CLI changes introduced in OneFS 9.5 are as follows:

Operation OneFS 9.5 Change Details
change-password Modified Needed to provide old password for changing so that we can calculate how many chars/percent changed
reset-password Added Generates a temp password that meets current password policy for user to login
set-password Deprecated Doesn’t need to provide old password

A user’s password can now be set, changed, and reset by either ‘root’ or ‘admin’. This is supported by the new ‘isi auth users change-password’ or ‘isi auth users reset-password’ CLI command syntax. The latter, for example, returns a temporary password and requires the user to change it on next login. After logging in with the temporary (albeit secure) password, OneFS immediately forces the user to change it:

# whoami
admin

# isi auth users reset-password user1
4$_x\d\Q6V9E:sH

# ssh user1@localhost
(user1@localhost) Password:
(user1@localhost) Your password has expired.
You are required to immediately change your password.
Changing password for user1
New password:
(user1@localhost) Re-enter password:
Last login: Wed May 17 08:02:47 from 127.0.0.1
PowerScale OneFS 9.5.0.0

# whoami
user1

Also in OneFS 9.5 and later, the CLI ‘isi auth local view system’ command sees the addition of four new fields:

  • Password Chars Changed
  • Password Percent Changed
  • Password Hash Type
  • Max Inactivity Days

For example:

# isi auth local view system
                    Name: System
                  Status: active
          Authentication: Yes
   Create Home Directory: Yes
 Home Directory Template: /ifs/home/%U
        Lockout Duration: Now
       Lockout Threshold: 0
          Lockout Window: Now
             Login Shell: /bin/zsh
            Machine Name:
        Min Password Age: Now
        Max Password Age: 4W
     Min Password Length: 0
    Password Prompt Time: 2W
     Password Complexity: -
 Password History Length: 0
  Password Chars Changed: 0
Password Percent Changed: 0
      Password Hash Type: NTHash
     Max Inactivity Days: 0

The following CLI command syntax configures OneFS to require a minimum password length of 15 characters, a 50% or greater change, and 8 or more characters to be altered for a successful password reset:

# isi auth local modify system --min-password-length 15 --password-chars-changed 8 --password-percent-changed 50

Next, a command is issued to create a new user, ‘user2’, with a 10 character password:

# isi auth users create user2 --password 0123456789

Failed to add user user1: The specified password does not meet the configured password complexity or history requirements

This attempt fails because the password does not meet the configured password criteria (15 chars, 50% change, 8 chars to be altered).

Instead, the password for the new account, ‘user2’, is set to an appropriate value: ie  ‘0123456789abcdef’. Also, the ‘–prompt-password-change’ flag is included to force the user to change their password on next login.

# isi auth users create user2 --password 0123456789abcdef –prompt-password-change 1

On logging in to the ‘user2’ account, OneFS immediately prompts for a new password. In the example below, the following non-compliant password (‘012345678zyxw’) is entered.

  • 0123456789abcdef -> 012345678zyxw = Failure

This returns an unsuccessful change attempt failure, since it does not meet the 15 character minimum:

# su user2
New password:
Re-enter password:
The specified password does not meet the configured password complexity requirements.
Your password must meet the following requirements:
  * Must contain at least 15 characters.
  * Must change at least 8 characters.
  * Must change at least 50% of characters.
New password:

Instead, a compliant password and successful change could be:

  • 0123456789abcdef -> 0123456zyxwvuts = Success

The following command can also be used to change the password for a user. For example, to update user2’s password:

# isi auth users change-password user2
Current password (hit enter if none):
New password:
Confirm new password:

If a non-compliant password is entered, the following error is returned:

Password change failed: The specified password does not meet the configured password complexity or history requirements

When employed, OneFS hardening automatically enforces security-based configurations. The hardening engine is profile-based, and its STIG security profile is predicated on security mandates specified in the US Department of Defense (DoD) Security Requirements Guides (SRGs) and Security Technical Implementation Guides (STIGs).

On applying the STIG hardening security profile to a cluster (‘isi hardening apply –profile=STIG’), the password policy settings are automatically reconfigured to the following values:

Field Normal Value STIG Hardened
Lockout Duration Now Now
Lockout Threshold 0 3
Lockout Window Now 15m
Min Password Age Now 1D
Max Password Age 4W 8W4D
Min Password Length 0 15
Password Prompt Time 2W 2W
Password Complexity lowercase, numeric, repeat, symbol, uppercase
Password History Length 0 5
Password Chars Changed 0 8
Password Percent Changed 0 50
Password Hash Type NTHash SHA512
Max Inactivity Days 0 35

For example:

# uname -or
Isilon OneFS 9.5.0.0

# isi hardening list
Name  Description                       Status
---------------------------------------------------
STIG  Enable all STIG security settings Applied
---------------------------------------------------
Total: 1

# isi auth local view system
                    Name: System
                  Status: active
          Authentication: Yes
   Create Home Directory: Yes
 Home Directory Template: /ifs/home/%U
        Lockout Duration: Now
       Lockout Threshold: 3
          Lockout Window: 15m
             Login Shell: /bin/zsh
            Machine Name:
        Min Password Age: 1D
        Max Password Age: 8W4D
     Min Password Length: 15
    Password Prompt Time: 2W
     Password Complexity: lowercase, numeric, repeat, symbol, uppercase
 Password History Length: 5
  Password Chars Changed: 8
Password Percent Changed: 50
      Password Hash Type: SHA512
     Max Inactivity Days: 35

Note that the ‘Password Hash Type’ is changed from the default ‘NTHash’ to the more secure ‘SHA512’ encoding, in addition to setting the various password criteria.

The OneFS 9.5 WebUI also sees several additions and alterations to the Password Policy page. These include:

Operation OneFS 9.5 Change Details
Policy page Added New Password policy page under Access -> Membership and Roles
reset-password Added Generate a random password that meets current password policy for user to login

The most obvious change is the transfer of the policy configuration elements from the local provider page to a new dedicated Password Policy page.

Here’s the OneFS 9.4 ‘View a local provider’ page, under Access > Authentication providers > Local providers > System:

The above is replaced and augmented in the OneFS 9.5 WebUI with the following page, located under Access > Membership and Roles > Password Policy:

New password policy configuration options are included to require upper-case, lower-case, numeric, or special characters and limit the number of contiguous repeats of a character, etc.

When it comes to changing a password, only a permitted user can make their change. This can be performed from a couple of locations in the WebUI. First, the user options on the task bar at the top of each screen now provides a ‘Change password’ option:

A pop-up warning message will also be displayed by the WebUI, informing when password expiration is imminent. This warning provides a ‘Change Password’ link:

Clicking on the above link displays the following page:

A new password complexity tool-tip message is also displayed, informing the user of safe password selection.

Note that re-login is required after a password change.

On the ‘Users’ page under Access > Membership and roles > Users, the ‘Action’ drop-down list on the ‘Users’ page now also contains a ‘Reset Password’ option:

The successful reset confirmation pop-up offers both a ‘show’ and ‘copy’ option, while informing the cluster administrator to share the new password with the user, and for them to change their password during their next login:

The ‘Create user’ page now provides an additional field that requires password confirmation. Additionally, the password complexity tool-tip message is also displayed:

The redesigned ‘Edit user details’ page no longer provides a field to edit the password directly:

Instead, the ‘Action’ drop-down list on the ‘Users’ page now contains a ‘Reset Password’ option.

OneFS Key Manager Rekey Support

The OneFS key manager is a backend service which orchestrates the storage of sensitive information for PowerScale clusters. In order to satisfy Dell’s Secure Infrastructure Ready requirements and other public and private sector security mandates, the manager provides the ability to replace, or rekey, cryptographic keys.

The quintessential consumer of OneFS key management is data-at-rest encryption (DARE). Protecting sensitive data stored on the cluster with cryptography ensures that it’s guarded against theft, in the event that drives or nodes are removed from a PowerScale cluster. DARE is a requirement for federal and industry regulations ensuring data is encrypted when it is stored. OneFS has provided DARE solutions for many years through secure encrypted drives (SED) and the OneFS key management system.

A 256-bit Master Key (MK) encrypts the Key Manager Database (KMDB) for SED and cluster domains. In OneFS 9.2 and later, the MK for SEDs can either be stored off-cluster on a KMIP server, or locally on a node (the legacy behavior).

However, there are a variety of other consumers of the OneFS key manager, in addition to DARE. These include services and protocols such as:

Service Description
CELOG Cluster event log.
CloudPools Cluster tier to cloud service.
Email Electronic mail.
FTP File transfer protocol.
IPMI Intelligent platform management interface for remote cluster console access.
JWT JSON web tokens.
NDMP Network data management protocol for cluster backups and DR.
Pstore Active directory and Kerberos password store.
S3 S3 object protocol.
SyncIQ Cluster replication service.
SmartSync OneFS push and pull replication cluster and cloud replication service.
SNMP Simple network monitoring protocol.
SRS Old Dell support remote cluster connectivity.
SSO Single sign-on.
SupportAssist Remote cluster connectivity to Dell Support.

OneFS 9.5 introduces a number of enhancements to the venerable key manager, including:

  • The ability to rekey keystores. Rekey operation will generate a new MK and re-encrypt all entries stored with the new key.
  • New CLI commands and WebUI options to perform a rekey operation or schedule key rotation on a time interval.
  • New commands to monitor the progress and status of a rekey operation.

As such, OneFS 9.5 now provides the ability to rekey the MK, irrespective of where it is stored.

Note that, when upgrading from an earlier OneFS release, the new rekey functionality is only available once the OneFS 9.5 upgrade has been committed.

Under the hood, each provider store in the key manager consists of secure backend storage and an MK. Entries are kept in a Sqlite database or key-value store. A provider datastore uses its MK to encrypt all its entries within the store.

During the rekey process, the old MK is only deleted after a successful re-encryption with the new MK. If, for any reason, the process fails, the old MK is available and remains as the current MK. The rekey daemon retries the rekey every 15 minutes if the process fails.

The OneFS rekey process is as follows:

  1. A new master key (MK) is generated, and internal configuration is updated.
  2. Any entries in the provider store are decrypted and encrypted with the new MK.
  3. If the prior steps are successful, the previous MK is deleted

To support the rekey process, the MK in OneFS 9.5 now has an ID associated with it. All entries have a new field referencing the master key ID.

During the rekey operation, there are two MK values with different IDs, and all entries in the database will associate which key they are encrypted by.

In OneFS 9.5, the rekey configuration and management is split between the cluster keys and the SED keys:

Rekey Component Detail
SED ·         SED provider keystore is stored locally on each node.

·         SED provider domain already had an existing CLI commands for handling KMIP settings in prior releases.

Cluster ·         Controls all cluster-wide keystore domains.

·         Status shows information of all cluster provider domains.

SED keys rekey

The SED key manager rekey operation can be managed through a DARE cluster’s CLI or WebUI, and can either be automatically scheduled or run manually on-demand. The following CLI syntax can be used to manually initiate a rekey:

# isi keymanager sed rekey start

Alternatively, to schedule a rekey operation. For example, to schedule a key rotation every two months:

# isi keymanager sed rekey modify --key-rotation 2M

The key manager status for SEDs can be viewed as follows:

# isi keymanager sed status

 Node Status  Location  Remote Key ID  Key Creation Date   Error Info(if any)

-----------------------------------------------------------------------------

1   LOCAL   Local                    1970-01-01T00:00:00

-----------------------------------------------------------------------------

Total: 1

Alternatively, from the WebUI, navigate to Access > Key Management >  SED/Cluster Rekey and check the ‘Automatic rekey for SED keys’ and configure the rekey frequency:

Note that for SED rekey operations, if a migration from local cluster key management to a KMIP server is in progress, the rekey process will begin once the migration has completed.

Cluster keys rekey

As mentioned previously, OneFS 9.5 also supports the rekey of cluster keystore domains. This cluster rekey operation is available through the CLI and the WebUI and may either be scheduled or run on-demand. The available cluster domains can be queried by running the following CLI syntax:

# isi keymanager cluster status

Domain     Status  Key Creation Date   Error Info(if any)

----------------------------------------------------------

CELOG      ACTIVE  2023-04-06T09:19:16

CERTSTORE  ACTIVE  2023-04-06T09:19:16

CLOUDPOOLS ACTIVE  2023-04-06T09:19:16

EMAIL      ACTIVE  2023-04-06T09:19:16

FTP        ACTIVE  2023-04-06T09:19:16

IPMI_MGMT  IN_PROGRESS  2023-04-06T09:19:16

JWT        ACTIVE  2023-04-06T09:19:16

LHOTSE     ACTIVE  2023-04-06T09:19:11

NDMP       ACTIVE  2023-04-06T09:19:16

NETWORK    ACTIVE  2023-04-06T09:19:16

PSTORE     ACTIVE  2023-04-06T09:19:16

RICE       ACTIVE  2023-04-06T09:19:16

S3         ACTIVE  2023-04-06T09:19:16

SIQ        ACTIVE  2023-04-06T09:19:16

SNMP       ACTIVE  2023-04-06T09:19:16

SRS        ACTIVE  2023-04-06T09:19:16

SSO        ACTIVE  2023-04-06T09:19:16

----------------------------------------------------------

Total: 17

The rekey process generates a new key and re-encrypts the entries for the domain. The old key is then deleted.

Performance-wise, the rekey process does consume cluster resources (CPU and disk) as a result of the re-encryption phase which is fairly write-intensive. As such, a good practice is to perform rekey operations outside of core business hours, or during scheduled cluster maintenance windows.

During the rekey process, the old MK is only deleted oncer a successful re-encryption with the new MK has been confirmed. In the event of a rekey process failure, the old MK is available and remains as the current MK.

A rekey may be requested immediately or may be scheduled with a cadence. The rekey operation is available through the CLI and the WebUI. In the WebUI, navigate to Access > Key Management > SED/Cluster Rekey.

To start a rekey of the cluster domains immediately, from the CLI run the following syntax:

# isi keymanager cluster rekey start
Are you sure you want to rekey the master passphrase? (yes/[no]):yes

Alternatively, from the WebUI, navigate to Access under the SED/Cluster Rekey tab, select Rekey Now button, next to Cluster Keys:

A scheduled rekey of the cluster keys (excluding the SED keys) can be configured from the CLI with the following syntax:

# isi keymanager cluster rekey modify --key-rotation [YMWDhms].

Specify the frequency of the ‘key rotation’ field as an integer, using Y for years, M for months, W for weeks, D for days, h for hours, m for minutes, and s for seconds. For example, the following command will schedule the cluster rekey operation to execute every six weeks:

# isi keymanager cluster rekey view
Rekey Time: 1970-01-01T00:00:00
Key Rotation: Never
# isi keymanager cluster rekey modify --key-rotation 6W
# isi keymanager cluster rekey view
Rekey Time: 2023-04-28T18:38:45
Key Rotation: 6W

The rekey configuration can be easily reverted back to on-demand from a schedule as follows:

# isi keymanager cluster rekey modify --key-rotation Never
# isi keymanager cluster rekey view
Rekey Time: 2023-04-28T18:38:45
Key Rotation: Never

Alternatively, from the WebUI, under the SED/Cluster Rekey tab, select the Automatic rekey for Cluster keys checkbox and specify the rekey frequency. For example:

In an event of a rekeying failure a CELOG ‘KeyManagerRekeyFailed’ or ‘KeyManagerSedsRekeyFailed’ event is created. Since SED rekey is a node-local operation, the ‘KeyManagerSedsRekeyFailed’ event information will also include which node experienced the failure.

Additionally, current cluster rekey status can also be queried with the following CLI command:

# isi keymanager cluster status

Domain     Status  Key Creation Date   Error Info(if any)

----------------------------------------------------------

CELOG      ACTIVE  2023-04-06T09:19:16

CERTSTORE  ACTIVE  2023-04-06T09:19:16

CLOUDPOOLS ACTIVE  2023-04-06T09:19:16

EMAIL      ACTIVE  2023-04-06T09:19:16

FTP        ACTIVE  2023-04-06T09:19:16

IPMI_MGMT  ACTIVE  2023-04-06T09:19:16

JWT        ACTIVE  2023-04-06T09:19:16

LHOTSE     ACTIVE  2023-04-06T09:19:11

NDMP       ACTIVE  2023-04-06T09:19:16

NETWORK    ACTIVE  2023-04-06T09:19:16

PSTORE     ACTIVE  2023-04-06T09:19:16

RICE       ACTIVE  2023-04-06T09:19:16

S3         ACTIVE  2023-04-06T09:19:16

SIQ        ACTIVE  2023-04-06T09:19:16

SNMP       ACTIVE  2023-04-06T09:19:16

SRS        ACTIVE  2023-04-06T09:19:16

SSO        ACTIVE  2023-04-06T09:19:16

----------------------------------------------------------

Total: 17

Or, for SEDs rekey status:

# isi keymanager sed status

 Node Status  Location  Remote Key ID  Key Creation Date   Error Info(if any)

-----------------------------------------------------------------------------

1   LOCAL   Local                    1970-01-01T00:00:00

2   LOCAL   Local                    1970-01-01T00:00:00

3   LOCAL   Local                    1970-01-01T00:00:00

4   LOCAL   Local                    1970-01-01T00:00:00

-----------------------------------------------------------------------------

Total: 4

The rekey process also outputs to the /var/log/isi_km_d.log file, which is a useful source for additional troubleshooting.

If an error in rekey occurs, the previous MK is not deleted, so entries in the provider store can still be created and read as normal. The key manager daemon will re-attempt rekey operation in the background every fifteen minutes until it succeeds.

OneFS Restricted Shell – Log Viewing and Recovery

Complementary to the restricted shell itself, which was covered in the previous article in this series, OneFS 9.5 also sees the addition of a new log viewer, plus a recovery shell option.

The new isi_log_access CLI utility enables a secure shell user to read, page, and query the logfiles in the /var/log directory. The ability to run this tool is governed by user’s role being granted the ‘ISI_PRIV_SYS_SUPPORT’ role-based access control (RBAC) privilege.

OneFS RBAC is used to explicitly limit who has access to the range of cluster configurations and operations. This granular control allows administrative roles to be crafted which can create and manage the various OneFS core components and data services, isolating each to specific security roles or to admin only, etc.

In this case, a cluster security administrator selects the desired access zone, creates a zone-aware role within it, assigns the ‘ISI_PRIV_SYS_SUPPORT’ privileges, for isi_log_access use, and then assigns users to the role.

Note that the built-in OneFS ‘AuditAdmin’ RBAC role does not contains the ‘ISI_PRIV_SYS_SUPPORT’ by default. Also, the built-in RBAC roles cannot be reconfigured:

# isi auth roles modify AuditAdmin --add-priv=ISI_PRIV_SYS_SUPPORT

The privileges of built-in role AuditAdmin cannot be modified

Therefore, the ‘ISI_PRIV_SYS_SUPPORT’ role will need to be added to a custom role.

For example, the following CLI syntax will add the user ‘usr_admin_restricted’ to the ‘rl_ssh’ role, and add the privilege ‘ISI_PRIV_SYS_SUPPORT’ to the ‘rl_ssh’ role:

# isi auth roles modify rl_ssh --add-user=usr_admin_restricted

# isi auth roles modify rl_ssh --add-priv=ISI_PRIV_SYS_SUPPORT

# isi auth roles view rl_ssh
       Name: rl_ssh
Description: -
    Members: u_ssh_restricted
             u_admin_restricted
 Privileges
             ID: ISI_PRIV_LOGIN_SSH
     Permission: r

             ID: ISI_PRIV_SYS_SUPPORT
     Permission: r

The ‘usr_admin_restricted’ user can also be added to the ‘AuditAdmin’ role, if desired:

# isi auth roles modify AuditAdmin --add-user=usr_admin_restricted

# isi auth roles view AuditAdmin | grep -i member
    Members: usr_admin_restricted

So the isi_log_access tool itself supports the following command options and arguments:

Option Description
–grep Match a pattern against the file and display on stdout.
–help Displays the command description and usage message.
–list List all the files in the /var/log tree.
–less Display the file on stdout with a pager in secure_mode.
–more Display the file on stdout with a pager in secure_mode.
–view Display the file on stdout.
–watch Display the end of the file and new content as it is written.
–zgrep Match a pattern against the unzipped file contents and display on stdout.
–zview Display an unzipped version of the file on stdout.

 

Here the ‘u_admin_restricted’ user logs in to the secure shell and runs the isi_log_access utility to list the /var/log/messages logfile:

# ssh u_admin_restricted@10.246.178.121
(u_admin_restricted@10.246.178.121) Password:
Last login: Wed May  3 18:02:18 2023 from 10.246.159.107
Copyright (c) 2001-2023 Dell Inc. or its subsidiaries. All Rights Reserved.
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.

PowerScale OneFS 9.5.0.0

Allowed commands are
        clear ...
        isi ...
        isi_recovery_shell ...
        isi_log_access ...
        exit
        logout
# isi_log_access --list
LAST MODIFICATION TIME         SIZE       FILE
Mon Apr 10 14:22:18 2023       56         alert.log
Fri May  5 00:30:00 2023       62         all.log
Fri May  5 00:30:00 2023       99         all.log.0.gz
Fri May  5 00:00:00 2023       106        all.log.1.gz
Thu May  4 00:30:00 2023       100        all.log.2.gz
Thu May  4 00:00:00 2023       107        all.log.3.gz
Wed May  3 00:30:00 2023       99         all.log.4.gz
Wed May  3 00:00:00 2023       107        all.log.5.gz
Tue May  2 00:30:00 2023       100        all.log.6.gz
Mon Apr 10 14:22:18 2023       56         audit_config.log
Mon Apr 10 14:22:18 2023       56         audit_protocol.log
Fri May  5 17:23:53 2023       82064      auth.log
Sat Apr 22 12:09:31 2023       10750      auth.log.0.gz
Mon Apr 10 15:31:36 2023       0          bam.log
Mon Apr 10 14:22:18 2023       56         boxend.log
Mon Apr 10 14:22:18 2023       56         bwt.log
Mon Apr 10 14:22:18 2023       56         cloud_interface.log
Mon Apr 10 14:22:18 2023       56         console.log
Fri May  5 18:20:32 2023       23769      cron
Fri May  5 15:30:00 2023       8803       cron.0.gz
Fri May  5 03:10:00 2023       9013       cron.1.gz
Thu May  4 15:00:00 2023       8847       cron.2.gz
Fri May  5 03:01:02 2023       3012       daily.log
Fri May  5 00:30:00 2023       101        daily.log.0.gz
Fri May  5 00:00:00 2023       1201       daily.log.1.gz
Thu May  4 00:30:00 2023       102        daily.log.2.gz
Thu May  4 00:00:00 2023       1637       daily.log.3.gz
Wed May  3 00:30:00 2023       101        daily.log.4.gz
Wed May  3 00:00:00 2023       1200       daily.log.5.gz
Tue May  2 00:30:00 2023       102        daily.log.6.gz
Mon Apr 10 14:22:18 2023       56         debug.log
Tue Apr 11 12:29:37 2023       3694       diskpools.log
Fri May  5 03:01:00 2023       244566     dmesg.today
Thu May  4 03:01:00 2023       244662     dmesg.yesterday
Tue Apr 11 11:49:32 2023       788        drive_purposing.log
Mon Apr 10 14:22:18 2023       56         ethmixer.log
Mon Apr 10 14:22:18 2023       56         gssd.log
Fri May  5 00:00:35 2023       41641      hardening.log
Mon Apr 10 15:31:05 2023       17996      hardening_engine.log
Mon Apr 10 14:22:18 2023       56         hdfs.log
Fri May  5 15:51:28 2023       31359      hw_ata.log
Fri May  5 15:51:28 2023       56527      hw_da.log
Mon Apr 10 14:22:18 2023       56         hw_nvd.log
Mon Apr 10 14:22:18 2023       56         idi.log

In addition to parsing an entire logfile with the ‘more’ and ‘less’ flags , the isi_log_access utility can also be used to watch (ie. ‘tail’) a log. For example, the /var/log/messages logfile:

% isi_log_access --watch messages
2023-05-03T18:00:12.233916-04:00 <1.5> h7001-2(id2) limited[68236]: Called ['/usr/bin/isi_log_access', 'messages'], which returned 2.
2023-05-03T18:00:23.759198-04:00 <1.5> h7001-2(id2) limited[68236]: Calling ['/usr/bin/isi_log_access'].
2023-05-03T18:00:23.797928-04:00 <1.5> h7001-2(id2) limited[68236]: Called ['/usr/bin/isi_log_access'], which returned 0.
2023-05-03T18:00:36.077093-04:00 <1.5> h7001-2(id2) limited[68236]: Calling ['/usr/bin/isi_log_access', '--help'].
2023-05-03T18:00:36.119688-04:00 <1.5> h7001-2(id2) limited[68236]: Called ['/usr/bin/isi_log_access', '--help'], which returned 0.
2023-05-03T18:02:14.545070-04:00 <1.5> h7001-2(id2) limited[68236]: Command not in list of allowed commands.
2023-05-03T18:02:50.384665-04:00 <1.5> h7001-2(id2) limited[68594]: Calling ['/usr/bin/isi_log_access', '--list'].
2023-05-03T18:02:50.440518-04:00 <1.5> h7001-2(id2) limited[68594]: Called ['/usr/bin/isi_log_access', '--list'], which returned 0.
2023-05-03T18:03:13.362411-04:00 <1.5> h7001-2(id2) limited[68594]: Command not in list of allowed commands.
2023-05-03T18:03:52.107538-04:00 <1.5> h7001-2(id2) limited[68738]: Calling ['/usr/bin/isi_log_access', '--watch', 'messages'].

As expected, the last few lines of the messages logfile are displayed. These log entries include the command audit entries for the ‘usr_admin_secure’ user running the ‘isi_log_access’ utility with both the ‘—-help’, ‘–list’, and ‘—-watch’ arguments.

The ‘isi_log_access’ utility also allows zipped logfiles to be read (–zview) or searched (–zgrep) without uncompressing them. For example, to find all the ‘usr_admin’ entries in the zipped vmlog.0.gz file:

# isi_log_access --zgrep usr_admin vmlog.0.gz
   0.0 64468 usr_admin_restricted /usr/local/bin/zsh
   0.0 64346 usr_admin_restricted python /usr/local/restricted_shell/bin/restricted_shell.py (python3.8)
   0.0 64468 usr_admin_restricted /usr/local/bin/zsh
   0.0 64346 usr_admin_restricted python /usr/local/restricted_shell/bin/restricted_shell.py (python3.8)
   0.0 64342 usr_admin_restricted sshd: usr_admin_restricted@pts/3 (sshd)
   0.0 64331 root               sshd: usr_admin_restricted [priv] (sshd)
   0.0 64468 usr_admin_restricted /usr/local/bin/zsh
   0.0 64346 usr_admin_restricted python /usr/local/restricted_shell/bin/restricted_shell.py (python3.8)
   0.0 64342 usr_admin_restricted sshd: usr_admin_restricted@pts/3 (sshd)
   0.0 64331 root               sshd: usr_admin_restricted [priv] (sshd)
   0.0 64468 usr_admin_restricted /usr/local/bin/zsh
   0.0 64346 usr_admin_restricted python /usr/local/restricted_shell/bin/restricted_shell.py (python3.8)
   0.0 64342 usr_admin_restricted sshd: usr_admin_restricted@pts/3 (sshd)
   0.0 64331 root               sshd: usr_admin_restricted [priv] (sshd)
   0.0 64468 usr_admin_restricted /usr/local/bin/zsh
   0.0 64346 usr_admin_restricted python /usr/local/restricted_shell/bin/restricted_shell.py (python3.8)
   0.0 64342 usr_admin_restricted sshd: u_admin_restricted@pts/3 (sshd)
   0.0 64331 root               sshd: usr_admin_restricted [priv] (sshd)

OneFS Recovery shell

The purpose of the recovery shell to allow a restricted shell user to access a regular UNIX shell, and its associated command set, if needed. As such, the recovery shell is primarily designed and intended for reactive cluster recovery operations, and other unforeseen support issues. Note that the ‘isi_recovery_shell’ CLI command can only be run, and the recovery shell entered, from within the restricted shell.

The ‘ISI_PRIV_RECOVERY_SHELL’ privilege is required in order for a user to elevate their shell from restricted to recovery. The following syntax can be used to add this privilege to a role, in this case the ‘rl_ssh’ role:

% isi auth roles modify rl_ssh --add-priv=ISI_PRIV_RECOVERY_SHELL

% isi auth roles view rl_ssh
       Name: rl_ssh
Description: -
    Members: usr_ssh_restricted
             usr_admin_restricted
 Privileges
             ID: ISI_PRIV_LOGIN_SSH
     Permission: r

             ID: ISI_PRIV_SYS_SUPPORT
     Permission: r

             ID: ISI_PRIV_RECOVERY_SHELL
     Permission: r

However, note that the ‘–-restricted-shell-enabled’ security parameter must be set to ‘true’ before a user with the ISI_PRIV_RECOVERY_SHELL privilege can actually enter the recovery shell. For example:

% isi security settings view | grep -i restr

Restricted shell Enabled: No

% isi security settings modify –restricted-shell-enabled=true

% isi security settings view | grep -i restr

Restricted shell Enabled: Yes

The restricted shell user will need to enter the cluster’s root password in order to successfully enter the recovery shell. For example:

% isi_recovery_shell -h
Description:
        This command is used to enter the Recovery shell i.e. normal zsh shell from the PowerScale Restricted shell. This command is supported only in the PowerScale Restricted shell.

Required Privilege:
        ISI_PRIV_RECOVERY_SHELL

Usage:
        isi_recovery_shell
           [{--help | -h}]

If root password is entered incorrectly, the following error will be displayed:

% isi_recovery_shell
Enter 'root' credentials to enter the Recovery shell
Password:
Invalid credentials.
isi_recovery_shell: PAM Auth Failed

A successful recovery shell launch is as follows:

$ ssh u_admin_restricted@10.246.178.121
(u_admin_restricted@10.246.178.121) Password:
Last login: Thu May  4 17:26:10 2023 from 10.246.159.107
Copyright (c) 2001-2023 Dell Inc. or its subsidiaries. All Rights Reserved.
Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.

PowerScale OneFS 9.5.0.0

Allowed commands are
        clear ...
        isi ...
        isi_recovery_shell ...
        isi_log_access ...
        exit
        logout

% isi_recovery_shell
Enter 'root' credentials to enter the Recovery shell
Password:
%

At this point, regular shell/UNIX commands (including the ‘vi’ editor) are available again:

% whoami
u_admin_restricted

% pwd
/ifs/home/u_admin_restricted
% top | head -n 10
last pid: 65044;  load averages:  0.12,  0.24,  0.29  up 24+04:17:23    18:38:39
118 processes: 1 running, 117 sleeping
CPU:  0.1% user,  0.0% nice,  0.9% system,  0.1% interrupt, 98.9% idle
Mem: 233M Active, 19G Inact, 2152K Laundry, 137G Wired, 60G Buf, 13G Free
Swap:
  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 3955 root          1 -22  r30    50M    14M select  24 142:28   0.54% isi_drive_d
 5715 root         20  20    0   231M    69M kqread   5  55:53   0.15% isi_stats_d
 3864 root         14  20    0    81M    21M kqread  16 133:02   0.10% isi_mcp

The specifics of the recovery shell (ZSH) for the u_admin_restricted user are reported as follows:

% printenv $SHELL
_=/usr/bin/printenv
PAGER=less
SAVEHIST=2000
HISTFILE=/ifs/home/u_admin_restricted/.zsh_history
HISTSIZE=1000
OLDPWD=/ifs/home/u_admin_restricted
PWD=/ifs/home/u_admin_restricted
SHLVL=1
LOGNAME=u_admin_restricted
HOME=/ifs/home/u_admin_restricted
RECOVERY_SHELL=TRUE
TERM=xterm
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

Shell logic conditions and scripts can be run. For example:

% while true; do uptime; sleep 5; done
 5:47PM  up 24 days,  3:26, 5 users, load averages: 0.44, 0.38, 0.34
 5:47PM  up 24 days,  3:26, 5 users, load averages: 0.41, 0.38, 0.34

ISI commands can be run and cluster management tasks performed.

% isi hardening list
Name  Description                       Status
---------------------------------------------------
STIG  Enable all STIG security settings Not Applied
---------------------------------------------------
Total: 1

For example, creating and deleting a snapshot:

% isi snap snap list
ID Name Path
------------
------------
Total: 0


% isi snap snap create /ifs/data

% isi snap snap list
ID   Name  Path
--------------------
2    s2    /ifs/data
--------------------
Total: 1

% isi snap snap delete 2
Are you sure? (yes/[no]): yes

Sysctls can be read and managed:

% sysctl efs.gmp.group

efs.gmp.group: <10539754> (4) :{ 1:0-14, 2:0-12,14,17, 3-4:0-14, smb: 1-4, nfs: 1-4, all_enabled_protocols: 1-4, isi_cbind_d: 1-4, lsass: 1-4, external_connectivity: 1-4 }

The restricted shell can be disabled:

% isi security settings modify --restricted-shell-enabled=false

% isi security settings view | grep -i restr
Restricted shell Enabled: No

However, the ‘isi underscore’ (isi_*) commands, such as isi_for_array, are still not permitted to run:

% /usr/bin/isi_for_array -s uptime
zsh: permission denied: /usr/bin/isi_for_array

% isi_gather_info
zsh: permission denied: isi_gather_info

% isi_cstats
isi_cstats: Syscall ifs_prefetch_lin() failed: Operation not permitted

When finished, the user can either end the session entirely with the ‘logout’ command, or quit the recovery shell via ‘exit’ and return to the restricted shell:

% exit

Allowed commands are

        clear ...
        isi ...
        isi_recovery_shell ...
        isi_log_access ...
        exit
        logout
%