Unstructured Data Quick Tips

When configuring security hardening on OneFS 9.5 or later, one thing to note is that, even with the STIG profile activated, not all the rules are automatically marked as ‘applied’. Specifically:

# isi hardening report view STIG | grep “Not Applied”

check_stig_celog_alerts                        Cluster   Not Applied Military Unique Deployment Guide manually configured CELOG settings.

check_synciq_default_ocsp_settings             Cluster   Not Applied /sync/settings/:cluster_certificate_id

check_synciq_policy_ocsp_settings              Cluster   Not Applied /sync/policies/:ocsp_issuer_certificate_id

check_multiple_ntp_servers_configured          Cluster   Not Applied /protocols/ntp/servers:total

set_auth_webui_sso_mfa_idp                     Cluster   Not Applied auth/providers/saml-services/idps/System

set_auth_webui_sso_mfa_sp_host                 Cluster   Not Applied auth/providers/saml-services/sp?zone=System:hostname

Applying a hardening profile is one of multiple tasks that are required in order to configure a STIG-compliant PowerScale cluster. These include:

Component	Tasks
Audit	Configure remote syslog servers for auditing.
Authentication	Configure secure auth provider, SecurityAdmin account, and default restricted shell.
CELOG	Create event channel for security officers and system admin to monitor /root and /var partition usage, audit service, security verification, and account creation.
MFA & SSO	Enable and configure multi-factor authentication and single sign-on.
NTP	Configure secure NTP servers with SHA256 keys.
SMB	Configure SMB global settings and defaults. Enable SMB encryption on shares.
SNMP	Enable SNMP and configure SNMPv3 settings.
SyncIQ	Configure SyncIQ to use CA certificates so both the source and target clusters (primary and secondary DSCs) have both Server Authentication and Client Authentication set in their Extended Key Usages fields.

Over the course of the next two blog articles, we’ll cover the specific configuration requirements and details of each of these components via the OneFS CLI.

In this article, we’ll focus on the following tasks:

Audit Setup

1. To set up secure auditing, first configure the remote syslog server(s). Note that, while the configuration differentiates between configuration, protocol, and system auditing, these can be sent to the same central syslog server(s). When complete, these syslog servers can be added to the OneFS audit configuration via the following CLI syntax:

# isi audit settings global modify --config-syslog-servers=[server FQDN/IP] --protocol-syslog-servers=[server FQDN/IP] --system-syslog-servers=[server FQDN/IP]

2. Also consider adding the cluster certificate to the audit settings for mutual Transport Layer Security (TLS) authentication.

# isi audit certificates syslog import [certificate_path] [key_path]

To prevent unauthorized access to the private key/certificate, the recommendation is to remove certificate and private key files once the necessary import steps have been completed.

Authentication Setup

1. Set the default shell for any new users created in the Local Provider.

# isi auth local modify System --login-shell=/usr/local/restricted_shell/bin/restricted_shell.py

2. Next, configure the remote authentication provider. This could be Kerberos, LDAP, or Active Directory. For more information, see the OneFS 9.5 CLI Administration Guide.

Note that all Active Directory users must have an e-mail address configured for them for use with ADFS multi-factor authentication (MFA).

Every Active Directory user must have a home directory created on the cluster, containing the correct public key in ~/.ssh/authorized_keys for the certificate presented by SSH clients (SecureCRT, PuTTY-CAC, etc).

If using Active Directory, the recommendation is to enable LDAP encryption, commonly referred to as ‘LDAP sign and seal’. For example:

# isi auth ads modify [provider-name] --ldap-sign-and-seal true

Additionally, the ‘machine password lifespan’ should be configured to a value of 60 days or less:

# isi auth ads modify [provider-name] --machine-password-lifespan=60D

Where [provider-name] is the name of the chosen Active Directory provider.

3. Finally, identify a remote-authenticated user and assign them administrative privileges.

# isi auth roles modify SecurityAdmin --add-user [username]

# isi auth roles modify SystemAdmin --add-user [username]

Where [username] is the name of the chosen administrative user.

CELOG Setup

1. For CELOG security setup, create and event channel for the required ISSO/SA alerts and configure appropriate event thresholds.

The following events need to send alerts on a channel monitored by an organization’s Information Systems Security Officers (ISSOs) or System Administrators (SAs):

Event ID	Event
100010001	The /var partition is near capacity.
100010002	The /var/crash partition is near capacity.
100010003	The root partition is near capacity.
400160002	Audit system cannot provide service.
400160005	Audit daemon failed to persist events.
400200001	Security verification check failed.
400200002	Security verification successfully ran.
400260000	User account(s) created/updated/removed.

The event channel can be created as follows:

# isi event channels create [channel name] [type] [options]

Next, the thresholds for the above event IDs can be set:

# isi event thresholds modify 100010001 --info 74 --warn 75

# isi event thresholds modify 100010002 --warn 75

# isi event thresholds modify 100010003 --warn 75

# isi event alerts create [event name 1] NEW [channel name]

 --eventgroup 100010001  --eventgroup 100010002 --eventgroup 100010003

 --eventgroup 400160002 --eventgroup 400160005 --eventgroup 400200001

 --eventgroup 400200002 --eventgroup 400260000



# isi event alerts create [event name 2] SEVERITY_INCREASE [channel name]

 --eventgroup 100010001 --eventgroup 100010002 --eventgroup 100010003

 --eventgroup 400160002 --eventgroup 400160005 --eventgroup 400200001

 --eventgroup 400200002 --eventgroup 400260000

Where:

Item	Description
[channel name]	The name of the newly configured event channel.
[event name 1] and [event name 2]	The names of the events that will trigger alerts when a new event occurs or when an event increases in severity, respectively.

Multi-Factor Authentication (MFA)/Single Sign-On (SSO) Setup

1. First, configure the SSO service provider. This can be achieved as follows:

# isi auth sso sp modify --hostname=[node IP or cluster FQDN]

Where [node IP or cluster FQDN] is the IP address of a node in the PowerScale cluster or the fully qualified domain name (FQDN) of the PowerScale cluster.

2. Next, configure the Identity Provider (IdP) as follows:

# isi auth sso idps create [name] [options]

3. Enable MFA/SSO.

# isi auth sso settings modify --sso-enabled=true

At this point, we’ve covered the configuration and setup of the first four components in the list.

In the next article in this series, we’ll focus on the remaining topics:

Namely secure NTP, SMB, SNMP, and SyncIQ configuration.

 

In the previous article, we took a look at the preparation and activation of OneFS security hardening. Now we turn out attention to its management and troubleshooting.

Once the STIG profile has been successfully activated, the bulk of the administrative attention is typically then focused on monitoring. However, applying a hardening profile is one of multiple steps needed to configure a truly STIG-compliant PowerScale cluster. These additional components include:

Component	Tasks
Audit	Configure remote syslog servers for auditing.
Authentication	Configure secure auth provider, SecurityAdmin account, and default restricted shell.
CELOG	Create event channel for security officers and system admin to monitor /root and /var partition usage, audit service, security verification, and account creation.
MFA	Enable and configure multi-factor authentication.
NTP	Configure secure NTP servers with SHA256 keys.
SMB	Configure SMB global settings and defaults. Enable SMB encryption on shares.
SNMP	Enable SNMP and configure SNMPv3 settings.
SSO	Configure single sign-on.
SyncIQ	Configure SyncIQ to use CA certificates so both the source and target clusters (primary and secondary DSCs) have both Server Authentication and Client Authentication set in their Extended Key Usages fields.

We will cover the above topics and related tasks in detail in the next article in this series.

When hardening is activated, the security hardening engine reads the STIG configuration from its config files. Sets of rules, or config items, are applied to the hardening configuration to increase security and/or ensure STIG compliance. These rules are grouped by profile, which contain collections of named rules. Profiles are now stored in separate .xml files under /etc/isi_hardening/profiles.

Currently there is just one profile available (STIG), but the infrastructure is in place to support additional profiles as an when they are required.

Similarly, the individual rules are stored in separate .xml files under /etc/isi_hardening/rules.

In OneFS 9.5 and later, these rules are grouped by functional area affected vs. release, and can now apply to platform API configuration ‘collections’. For example, a rule can be applied to all NFS exports or all SyncIQ policies. In addition to actionable rules, ‘check-only’ rules are supported which apply no changes.

As may be apparent by now, OneFS security hardening is currently managed from the CLI and platform API only. There is currently no WebUI area for hardening configuration.

Note that is strongly advised not to perform administrative actions until after STIG security profile activation has completed across all nodes in the cluster.

When it comes to troubleshooting the enablement and operation of a STIG hardening profile on a cluster, there are a cope of useful places to look.

The first step is to check the hardening report. In OneFS 9.5 and later, the hardening engine reporting infrastructure enables detailed reports to be generated that indicate which hardening rules are applied or not – as well as the cluster’s overall compliance status. For example, from the following non-compliant cluster:

# isi hardening reports create

...............Hardening operation complete.

# isi hardening reports list

Name  Applied  Status        Creation Date            Report Age

-----------------------------------------------------------------

STIG  No       Not Compliant Sat Apr 22 04:28:40 2023 2m1s

-----------------------------------------------------------------

Total: 1

# isi hardening reports view STIG | more

Name                              Location  Status      Setting

----------------------------------------------------------------------------------------------

logout_zsh_clear_screen           Node 1    Not Applied /etc/zlogout

logout_profile_clear_screen       Node 1    Not Applied /etc/profile

logout_csh_clear_screen           Node 1    Not Applied /etc/csh.logout

require_password_single_user_mode Node 1    Not Applied /etc/ttys

set_password_min_length_pam_01    Node 1    Not Applied /etc/pam.d/system

set_password_min_length_pam_02    Node 1    Not Applied /etc/pam.d/other

set_password_min_length_pam_03    Node 1    Not Applied /etc/pam.d/passwd

set_password_min_length_pam_04    Node 1    Not Applied /etc/pam.d/passwd

disable_apache_proxy              Node 1    Not Applied /etc/mcp/templates/isi_data_httpd.conf

disable_apache_proxy              Node 1    Not Applied /etc/mcp/templates/isi_data_httpd.conf

disable_apache_proxy              Node 1    Not Applied /etc/mcp/templates/isi_data_httpd.conf

set_shell_timeout_01              Node 1    Not Applied /etc/profile

set_shell_timeout_02              Node 1    Applied     /etc/zshrc

set_shell_timeout_03              Node 1    Not Applied /etc/zshrc

set_shell_timeout_04              Node 1    Not Applied /etc/csh.cshrc

set_dod_banner_02                 Node 1    Not Applied symlink:/etc/issue

check_node_default_umask          Node 1    Applied     umask

--More—(byte 2185)

As indicated in the truncated output above, the vast majority of the hardening elements on node 1 of this cluster have not been successfully applied. Note that these reports can be generated regardless of cluster hardening status.

Next, there are a couple of logfiles that can often yield useful clues and information:

• /var/log/hardening.log
• /var/log/hardening_engine.log

After scrutinizing these logfiles, manually running the hardening worker process and monitoring its output (stdout and stderr) is typically the next course of action.

The hardening service can be shut down if it’s running as follows:

# isi services -a | grep -i hard

   hardening_service    Hardening Service                        Enabled

# isi services -a hardening_service disable

# ps -auxw | grep -i hard

#

The following syntax can be used to manually run the hardening worker process on a node:

# /usr/bin/isi_hardening/hardening_worker.py --profile STIG --action <action>

In the above, ‘<action>’ can be one of ‘apply’, ‘defaults’, or ‘report_create’.

De-activating Hardening

After applying the STIG hardening profile to a OneFS 9.5 or later cluster, it is possible to re-apply the default (non-hardened) configuration with the following CLI syntax, which will undo the changes that hardening invoked. For example:

# isi hardening disable STIG

.........Hardening operation complete.

Note that with OneFS 9.5 and earlier, the ‘disable’ attempts to undo the effects of hardening, but does not guarantee a full restore of a prior cluster configuration.

This differs from the OneFS 9.4 and earlier, which uses the ‘isi hardening revert’ CLI command and process:

# isi hardening revert

Revert Started

This may take several minutes

……

Reverting Hardening profile successful

#

# isi hardening status

Cluster Name:  TME1

Hardening Status:  Not Hardened

Note that OneFS 9.5 security hardening does not support CEE (as well as ICAP and a number of other protocols). Specifically, CEE currently does not support TLS, and the STIG hardening profile disables non-TLS communications.

The STIG security profile In OneFS 9.5 and later automatically enables the host-based firewall, enforcing the specific ports and protocols required for the various cluster services. Ensure that the default OneFS port numbers and protocols do not conflict with any custom port or protocol configuration.

After the STIG security profile is activated on a cluster, certain STIG rules do not apply. For example, when a new user is added to the cluster, some values are system defaults rather than STIG defaults. After configuration changes are complete, reapply the STIG security profile for STIG defaults. Run a new hardening report to confirm if any parameters are no longer STIG compliant. The recommendation is to check the hardening report frequently, especially after running administrative commands, to detect whether any parameters are no longer STIG compliant.

Additionally, the following table lists the components and protocols that are not currently supported by OneFS 9.5 STIG hardening or do not meet FIPS compliance requirements:

Protocol / Component	Detail
SMB without encryption	Without SMB3 encryption enabled, the SMB protocol relies on weak cryptography, and does not meet FIPS requirements.
NFS without krb5p	The NFS protocol without Kerberos (krb5p) relies on weak cryptography, and does not meet FIPS requirements.
HDFS	The Hadoop HDFS protocol relies on weak cryptography, and does not meet FIPS requirements.
S3 via HTTP	The S3 protocol over HTTP relies on weak cryptography, and does not meet FIPS requirements.
CEE	Dell Common Event Enabler (CEE) relies on weak cryptography, and does not meet FIPS requirements.
ICAP/CAVA	ICAP and CAVA antivirus protocols rely on weak cryptography, and do not meet FIPS requirements.
NIS	NIS protocol relies on weak cryptography, and does not meet FIPS requirements.
SFTP	SFTP protocol lacks file audit capabilities.
SmartLock	A cluster configured for SmartLock compliance mode is incompatible with STIG hardening.

In the first article in this series, we took a look at the architecture and enhancements to security hardening in OneFS 9.5. Now we turn out attention to its preparation, configuration, and activation.

Applying a hardening profile is just one of multiple steps required in order to configure a STIG-compliant PowerScale cluster.

OneFS 9.5 security hardening comes pre-installed on a cluster, but not activated by default. Hardening is a licensed feature, and there are no changes to the licensing requirements or structure for OneFS 9.5 and later.

Applying a hardening profile is one of multiple steps required in order to configure a STIG-compliant PowerScale cluster. As such, the general process to apply and activate security hardening on a OneFS 9.5 or later cluster is as follows:

The specifics for each step are covered below:

1. Revert hardening on a cluster running OneFS 9.4 or earlier prior to upgrade.

Upgrading from a STIG-hardened OneFS 9.4 or earlier cluster to OneFS 9.5 and later is not supported:

Cluster Type	Upgrade Details
Non-hardened cluster	Upgrade to OneFS 9.5 on non-STIG hardened clusters is straightforward.
Hardened cluster	Upgrade from a STIG-hardened pre-OneFS 9.5 cluster to OneFS 9.5 is not supported. Revert cluster to a non-hardened state prior to upgrade to OneFS 9.5.

As such, if the cluster currently has hardening enabled, this must be reverted before upgrading to OneFS 9.5 or later.

To accomplish this, first, log in to the cluster’s CLI with a user account with the ‘ISI_PRIV_HARDENING’ RBAC role.

OneFS security hardening requires a license in order to be activated. If it is licensed, hardening can be applied as follows:

# isi hardening apply STIG
Apply Started
This may take several minutes
……
Applied Hardening profile successfully.
#

Once applied, hardening can be verified as follows:

# isi hardening status
Cluster Name:  TME1
Hardening Status:  Hardened
Profile:  STIG
Following is the nodewise status:
TME1-1 :  Enabled
TME1-2 :  Enabled
TME1-3 :  Enabled
TME1-4 :  Enabled

Hardening can be easily removed on clusters running OneFS 9.4 or earlier:

# isi hardening revert
Revert Started
This may take several minutes
……
Reverting Hardening profile successful
#
# isi hardening status
Cluster Name:  TME1
Hardening Status:  Not Hardened

2. Upgrade cluster to OneFS 9.5 or later.

The cluster must be running OneFS 9.5 or later in order to activate STIG hardening. If upgrading from an earlier release, the OneFS 9.5 or later upgrade must be committed before enabling hardening.

Upgrading a cluster on which security hardening has not been activated to OneFS 9.5 or later is straightforward and can be accomplished either by a simultaneous or rolling reboot strategy.

For example, to start a rolling upgrade, which is the default, run:

# isi upgrade cluster start <upgrade_image>

Similarly, the following CLI syntax will initiate a simultaneous upgrade:

# isi upgrade cluster start --simultaneous <upgrade_image>

Since OneFS supports the ability to roll back to the previous version, in-order to complete an upgrade it must be committed.

# isi upgrade cluster commit

The isi upgrade view CLI command can be used to monitor how the upgrade is progressing:

# isi upgrade view

Or, for an interactive session:

# isi upgrade view --interactive

3. Install hardening license.

To enable STIG hardening on versions prior to OneFS 9.5, first check that hardening is licensed on the cluster:

# isi license list | grep -i harden
HARDEN      4 Nodes     4 Nodes     Evaluation

A hardening license can be added as follows:

# isi license add <path_to_licenese_file>

Alternatively, a 90-day trial license can be activated on a lab/test cluster to evaluate STIG hardening:

# isi license add --evaluation HARDENING

If a current OneFS hardening license is not available when attempting to activate security hardening on a cluster, the following warning will be returned:

# isi hardening apply STIG

The HARDENING application is not currently installed. Please contact your Isilon account team for more information on evaluating and purchasing HARDENING.

4. Configure compliant password hash.

Before activating security hardening with the STIG profile, the password hash type should be set to use SHA512. For example:

# isi auth file modify System --password-hash-type=SHA512

NTLM support and authentication for all file protocols has been disabled for this provider due to change of password hash type.

# isi auth local modify System --password-hash-type=SHA512

Next, the account of last resort (ALR), which is ‘root’ on a PowerScale cluster, should be set to use this updated password hash type.

# isi auth users change-password root

If this step is skipped, attempts to apply hardening will fail with the following warning:

The hardening request was not accepted:

Account of last resort does not have a password set with a supported hash type (SHA256, SHA512): root.

The hardening profile was not applied.

Please see the Security Configuration Guide for guidance on how to set compatible account passwords.

The SSH key exchange algorithms should also be updated at this time with the following CLI syntax:

# isi ssh settings modify --kex-algorithms 'diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,ecdh-sha2-nistp384'

Finally, update the SSH ciphers as follows:

# isi ssh settings modify --ciphers 'aes256-ctr,aes256-gcm@openssh.com'

5. Activate STIG hardening.

The next step involves actually applying the STIG hardening profile. This can be accomplished as follows:

# isi hardening apply STIG
..............Hardening operation complete.

Note that password restrictions are only enforced for password changes that occur after applying hardening.

After applying the STIG hardening profile, it is possible to re-apply the default (non-hardened) configuration with the following CLI syntax, which will undo the changes that hardening invoked. For example:

# isi hardening disable STIG
.........Hardening operation complete.

Note that with OneFS 9.5 and earlier, the ‘disable’ attempts to undo the effects of hardening, but does not guarantee a full restore of a prior cluster configuration. This differs from the OneFS 9.4 and earlier hardening ‘isi hardening revert’ CLI command and process described in ‘step 1’ above.

6. Verify hardening configuration.

Finally, verify that the STIG hardening configuration was successful. This will be indicated by a status of ‘Applied’. For example:

# isi hardening list
Name  Description                       Status
-----------------------------------------------
STIG  Enable all STIG security settings Applied
-----------------------------------------------
Total: 1

Additionally, a report can be generated that provides a detailed listing of all the individual rules and their per-node status. For example:

# isi hardening report view STIG
logout_zsh_clear_screen           Node 1   Applied     /etc/zlogout                       
logout_profile_clear_screen       Node 1   Applied     /etc/profile                       
logout_csh_clear_screen           Node 1   Applied     /etc/csh.logout                    
require_password_single_user_mode Node 1   Applied     /etc/ttys                           
set_password_min_length_pam_01    Node 1   Applied     /etc/pam.d/system                  
set_password_min_length_pam_02    Node 1   Applied     /etc/pam.d/other                   
set_password_min_length_pam_03    Node 1   Applied     /etc/pam.d/passwd                  
set_password_min_length_pam_04    Node 1   Applied     /etc/pam.d/passwd                  
disable_apache_proxy              Node 1   Applied     /etc/mcp/templates/isi_data_httpd.conf
disable_apache_proxy              Node 1   Applied     /etc/mcp/templates/isi_data_httpd.conf
disable_apache_proxy              Node 1   Applied     /etc/mcp/templates/isi_data_httpd.conf
set_shell_timeout_01              Node 1   Applied     /etc/profile                       
set_shell_timeout_02              Node 1   Applied     /etc/zshrc                         
set_shell_timeout_03              Node 1   Applied     /etc/zshrc                         
set_shell_timeout_04              Node 1   Applied     /etc/csh.cshrc                      
set_dod_banner_02                 Node 1   Applied     symlink:/etc/issue                 
check_node_default_umask          Node 1   Applied     umask                              
set_celog_snmp_use_fips                        Cluster   Applied     N/A                   
disable_supportassist                          Cluster   Applied     -                     
disable_usb_ports                              Cluster   Applied     /security/settings:usb_ports_disabled
disable_ndmpd                                  Cluster   Applied     /protocols/ndmp/settings/global:service
enable_smtp_ssl                                Cluster   Applied     /1/cluster/email:smtp_auth_security
enable_onefs_cli                               Cluster   Applied     /security/settings:restricted_shell_enabled
set_min_password_percent_of_characters_changed Cluster   Applied     /16/auth/providers/local:password_percent_changed
set_ads_ldap_sign_and_seal                     Cluster   Applied     -                     
set_ads_ldap_sign_and_seal_default             Cluster   Applied     registry.Services.lsass.Parameters.Providers.ActiveDirectory.LdapSignAndSeal
set_ads_machine_password_changes               Cluster   Applied     -                     
limit_ads_machine_password_lifespan            Cluster   Applied     -                     
enable_firewall                                Cluster   Applied     /network/firewall/settings:enabled
disable_audit_log_delete                       Cluster   Applied     /ifs/.ifsvar/audit/log_delete
set_audit_retention_period                     Cluster   Applied     /audit/settings/global:retention_period
disable_webui_access_ran                       Cluster   Applied     webui_ran_access      
set_ssh_config_client_alive_interval           Cluster   Applied     client_alive_interval 
set_ssh_config_client_alive_count              Cluster   Applied     client_alive_count_max
set_nfs_security_flavors                       Cluster   Applied     /protocols/nfs/exports:security_flavors
set_nfs_security_flavors                       Cluster   Applied     /protocols/nfs/exports:security_flavors
set_nfs_security_flavors                       Cluster   Applied     /protocols/nfs/exports:security_flavors
set_nfs_security_flavors                       Cluster   Applied     /protocols/nfs/exports:security_flavors
set_nfs_security_flavors                       Cluster   Applied     /protocols/nfs/exports:security_flavors
set_nfs_default_security_flavors               Cluster   Applied     /protocols/nfs/settings/export:security_flavors
set_nfs_default_security_flavors               Cluster   Applied     /protocols/nfs/settings/export:security_flavors
set_nfs_default_security_flavors               Cluster   Applied     /protocols/nfs/settings/export:security_flavors
set_nfs_default_security_flavors               Cluster   Applied     /protocols/nfs/settings/export:security_flavors
set_nfs_default_security_flavors               Cluster   Applied     /protocols/nfs/settings/export:security_flavors
set_s3_https_only                              Cluster   Applied     /protocols/s3/settings/global:https_only
check_ipmi_enabled                             Cluster   Applied     -                      
set_cnsa_crypto_http                           Cluster   Applied     cipher_suites         
set_cnsa_crypto_webui                          Cluster   Applied     cipher_suites         
disable_hdfs                                   Cluster   Applied     registry.Services.lsass.Parameters.Zones.System.HdfsEnabled
disable_webhdfs                                Cluster   Applied     registry.Services.lsass.Parameters.Zones.System.WebHdfsEnabled
disable_http_basic_authentication              Cluster   Applied     /protocols/http/settings:basic_authentication
disable_http_dav                               Cluster   Applied     /protocols/http/settings:dav
enable_http_integrated_authentication          Cluster   Applied     /protocols/http/settings:integrated_authentication
set_apache_loglevel                            Cluster   Applied     log_level              
set_apache_inactive_timeout                    Cluster   Applied     /protocols/http/settings:inactive_timeout
set_apache_session_max_age                     Cluster   Applied     /protocols/http/settings:session_max_age
disable_cee                                    Cluster   Applied     /audit/settings/global:cee_server_uris
check_stig_celog_alerts                        Cluster   Not Applied Military Unique Deployment Guide manually configured CELOG settings.
set_auth_concurrent_session_limit              Cluster   Applied     16/auth/settings/global:concurrent_session_limit
set_ldap_tls_revocation_check_level            Cluster   Applied     -                     
set_ldap_default_tls_revocation_check_level    Cluster   Applied     /auth/settings/global:default_ldap_tls_revocation_check_level
set_synciq_require_encryption                  Cluster   Applied     14/sync/settings:encryption_required
check_synciq_default_ocsp_settings             Cluster   Not Applied /sync/settings/:cluster_certificate_id
check_synciq_policy_ocsp_settings              Cluster   Not Applied /sync/policies/:ocsp_issuer_certificate_id
check_daemon_user_disabled                     Cluster   Applied     /auth/users/USER:daemon/:enabled
check_multiple_ntp_servers_configured          Cluster   Not Applied /protocols/ntp/servers:total
check_celog_smtp_channels_use_tls              Cluster   Applied     -                     
set_apache_service_timeout                     Cluster   Applied     /protocols/http/settings:service_timeout
set_dm_tls_revocation_check_level              Cluster   Applied     /datamover/certificates/settings/:revocation_setting
check_one_account_of_last_resort               Cluster   Applied     Number of UID:0 accounts configured
set_krb5_default_tgs_enctypes                  Cluster   Applied     /auth/settings/krb5/defaults:default_tgs_enctypes
set_krb5_default_tkt_enctypes                  Cluster   Applied     /auth/settings/krb5/defaults:default_tkt_enctypes
set_krb5_permitted_enctypes                    Cluster   Applied     /auth/settings/krb5/defaults:permitted_enctypes
set_krb5_preferred_enctypes                    Cluster   Applied     /auth/settings/krb5/defaults:preferred_enctypes
set_local_lockouts_duration                    Cluster   Applied     /auth/providers/local/:lockout_duration
set_local_lockouts_threshold                   Cluster   Applied     /auth/providers/local/:lockout_threshold
set_local_lockouts_window                      Cluster   Applied     /auth/providers/local/:lockout_window
set_local_max_password_age                     Cluster   Applied     /auth/providers/local/:max_password_age
set_local_min_password_age                     Cluster   Applied     /auth/providers/local/:min_password_age
set_local_password_chars_changed               Cluster   Applied     /auth/providers/local/:min_password_length
set_local_max_inactivity                       Cluster   Applied     /auth/providers/local/:max_inactivity_days
set_global_failed_login_delay                  Cluster   Applied     /auth/settings/global:failed_login_delay_time
set_ldap_require_secure_connection             Cluster   Applied     -                     
set_ldap_do_not_ignore_tls_errors              Cluster   Applied     -                     
set_ldap_tls_protocol_min_version              Cluster   Applied     -                     
set_ldap_ntlm_support                          Cluster   Applied     -                     
disable_nis                                    Cluster   Applied     -                     
disable_duo                                    Cluster   Applied     /auth/providers/duo/:enabled
set_ntlm_support_file                          Cluster   Applied     /auth/providers/file/:ntlm_support
check_password_hashes                          Cluster   Applied     lsa-file-provider:System:root password hash
set_file_enabled                               Cluster   Applied     /auth/users/<USER>:enabled
set_local_disabled_when_inactive               Cluster   Applied     /auth/users/<USER>:disabled_when_inactive
set_local_disabled_when_inactive_default       Cluster   Applied     registry.Services.lsass.Parameters.Providers.Local.DefaultDisableWhenInactive
set_auth_webui_sso_mfa_enabled                 Cluster   Applied     auth/providers/saml-services/settings?zone=System:sso_enabled
set_auth_webui_sso_mfa_idp                     Cluster   Not Applied auth/providers/saml-services/idps/System
set_auth_webui_sso_mfa_sp_host                 Cluster   Not Applied auth/providers/saml-services/sp?zone=System:hostname
set_auth_webui_sso_mfa_required                Cluster   Applied     authentication_mode   
disable_remotesupport                          Cluster   Applied     /auth/users/USER:remotesupport/:enabled
enable_audit_1                                 Cluster   Applied     /audit/settings/global:protocol_auditing_enabled
enable_audit_2                                 Cluster   Applied     /audit/settings:syslog_forwarding_enabled
disable_vsftpd                                 Cluster   Applied     /protocols/ftp/settings:service
disable_snmpv1_v2                              Cluster   Applied     5/protocols/snmp/settings:snmp_v1_v2c_access
set_snmp_v3_auth_protocol_sha                  Cluster   Applied     5/protocols/snmp/settings:snmp_v3_auth_protocol
disable_srs                                    Cluster   Applied     /esrs/status:enabled  
set_password_min_length                        Cluster   Applied     /auth/providers/local/:min_password_length
set_min_password_complexity                    Cluster   Applied     /auth/providers/local/:password_complexity
set_password_require_history                   Cluster   Applied     /auth/providers/local/:password_history_length
disable_coredump_minidump                      Cluster   Applied     /etc/mcp/templates/sysctl.conf
set_dod_banner_01                              Cluster   Applied     /cluster/identity:motd_header
set_listen_on_ip_controlpath                   Cluster   Applied     listen_on_ip           
set_listen_on_ip_datapath                      Cluster   Applied     listen_on_ip          
enable_fips_mode                               Cluster   Applied     /security/settings:fips_mode_enabled
disable_kdb                                    Cluster   Applied     /etc/mcp/templates/sysctl.conf
disable_basic_auth                             Cluster   Applied     auth_basic            
disable_cava                                   Cluster   Applied     /avscan/settings:service_enabled
require_smb3_encryption                        Cluster   Applied     /protocols/smb/settings/global:support_smb1

Over the course of the last few months, the topics for these blog articles have primarily focused on cluster security, and the variety of supporting features and enhancements that OneFS 9.5 introduced to this end. These include:

Component	Enhancement
Cryptography	FIPS 140-2 data-in-flight encryption for major protocols, FIPS 140-2 data at rest through SEDs, SEDs master key rekey, and TLS 1.2 support.
Public Key Infrastructure	Common Public Key Infrastructure (PKI) library, providing digital signature and encryption capabilities.
Certificates	PKI to issue, maintain, and revoke public key certificates.
Firewall	Host-based firewall, permitting restriction of the management interface to a dedicated subnet and hosts to specified IP pools.
Audit	OneFS system configuration auditing via CEE.
Authentication	Multifactor authentication (MFA), single sign-on (SSO) through SAML for the WebUI, and PKI-based authentication.
HTTP	HTTP Service Separation.
IPv6	IPV6-only network support for the USGv6R1 standard.
Restricted Shell	Secure shell with limited access to cluster command line utilities. Eliminates areas where commands and scripts could be run and files modified maliciously and unaudited.

While these features and tools can be activated, configured, and controlled manually, they can also be enabled automatically by a OneFS security policy, under the purview of the OneFS Hardening Engine.

While security hardening has been a salient part of OneFS since 7.2.1, the underlying infrastructure saw a significant redesign and augmentation in OneFS 9.5. A primary motivation for this overhaul was to comply with the current stringent US Federal security mandates and ready PowerScale for inclusion in the Department of Defense Information Networks (DoDIN) Approved Product List (APL). Specifically, compliance with the applicable DoD Security Requirements Guides (SRGs) and Security Technical Implementation Guides (STIGs).

While retaining its legacy functionality, the enhanced security hardening functionality in OneFS 9.5 enhances both scope, scale, and accountability.  The basic hardening architecture is as follows:

When hardening is activated, the security hardening engine reads the STIG configuration from its config files. Sets of rules, or config items, are applied to the hardening configuration to increase security and/or ensure STIG compliance. These rules are grouped by profile, which contain collections of named rules. Profiles are now stored in separate .xml files under /etc/isi_hardening/profiles.

# ls /etc/isi_hardening/profiles

profile_stig.xml

Currently there is just one profile available (STIG), but the infrastructure is in place to support additional profiles as an when they are required. As of OneFS 9,5, the STIG profile contains over 100 rules.

Similarly, the individual rules are stored in separate .xml files under /etc/isi_hardening/rules.

# ls /etc/isi_hardening/rules

rules_apache.xml        rules_celog.xml         rules_pki_ocsp.xml

rules_audit.xml         rules_fips.xml          rules_shell_timeout.xml

rules_auth.xml          rules_misc.xml          rules_umask.xml

rules_banners.xml       rules_password.xml

In OneFS 9.5 and later, these rules are grouped by functional area affected (as opposed to by release in earlier versions), and can now apply to platform API configuration ‘collections’. For example, a rule can be applied to all NFS exports or all SyncIQ policies. In addition to actionable rules, ‘check-only’ rules are supported which apply no changes.

The new OneFS 9.5 rules are also smarter, and now allow comparator logic in addition to the previous equality. For example, the new rules can evaluate conditions like whether a string is empty or non-empty, and if a given timeout is greater or equal to the required value.

Examples of STIG hardening rules include:

Functional Area	Rule Description
Firewall	Enables the OneFS firewall.
WebUI	Forces the OneFS WebUI to listen on a specific IP address.
Restricted Shell	Enforces the use of the restricted shell.
WebDAV	Disables WebDAV HTTP filesystem access.
SyncIQ	Enabling encrypted transport for all SyncIQ replication policies.

For example:

# cat /etc/isi_hardening/profile_stig.xml

<?xml version="1.0" encoding="UTF-8"?>

<Profiles version="1">

    <Profile>

        <Name>STIG</Name>

        <Description>Enable all STIG security settings</Description>

        <Rule>set_celog_snmp_use_fips</Rule>

        <Rule>disable_supportassist</Rule>

        <Rule>disable_usb_ports</Rule>

        <Rule>disable_ndmpd</Rule>

        <Rule>enable_smtp_ssl</Rule>

        <Rule>enable_onefs_cli</Rule>

        <Rule>set_min_password_percent_of_characters_changed</Rule>

        <Rule>set_ads_ldap_sign_and_seal</Rule>

        <Rule>set_ads_ldap_sign_and_seal_default</Rule>

        <Rule>set_ads_machine_password_changes</Rule>

        <Rule>limit_ads_machine_password_lifespan</Rule>

        <Rule>enable_firewall</Rule>

        <Rule>disable_audit_log_delete</Rule>

        <Rule>set_audit_retention_period</Rule>

        <Rule>disable_webui_access_ran</Rule>

        <Rule>set_ssh_config_client_alive_interval</Rule>

        <Rule>set_ssh_config_client_alive_count</Rule>

        <Rule>set_nfs_security_flavors</Rule>

<Snip>

Several enhancements have been made to the hardening engine in OneFS 9.5, the most notable of which is a significant increase in the number of rules permitted. The hardening engine also now includes a reporting component, allowing detailed reports to be generated that indicate which hardening rules are applied or not, as well as overall compliance status.  For example:

# isi hardening reports create

...............Hardening operation complete.

# isi hardening reports list

Name  Applied  Status        Creation Date            Report Age

-----------------------------------------------------------------

STIG  No       Compliant Sat Apr 22 04:28:40 2023 2m1s

-----------------------------------------------------------------

Total: 1

# isi hardening reports view STIG | more

Name                              Location  Status      Setting

----------------------------------------------------------------------------------------------

logout_zsh_clear_screen           Node 8    Applied /etc/zlogout

logout_profile_clear_screen       Node 8    Applied /etc/profile

logout_csh_clear_screen           Node 8    Applied /etc/csh.logout

require_password_single_user_mode Node 8    Not Applied /etc/ttys

set_password_min_length_pam_01    Node 8    Not Applied /etc/pam.d/system

set_password_min_length_pam_02    Node 8    Not Applied /etc/pam.d/other

set_password_min_length_pam_03    Node 8    Not Applied /etc/pam.d/passwd

set_password_min_length_pam_04    Node 8    Not Applied /etc/pam.d/passwd

disable_apache_proxy              Node 8    Not Applied /etc/mcp/templates/isi_data_httpd.conf

disable_apache_proxy              Node 8    Not Applied /etc/mcp/templates/isi_data_httpd.conf

disable_apache_proxy              Node 8    Not Applied /etc/mcp/templates/isi_data_httpd.conf

set_shell_timeout_01              Node 8    Not Applied /etc/profile

set_shell_timeout_02              Node 8    Applied     /etc/zshrc

set_shell_timeout_03              Node 8    Not Applied /etc/zshrc

set_shell_timeout_04              Node 8    Not Applied /etc/csh.cshrc

set_dod_banner_02                 Node 8    Not Applied symlink:/etc/issue

check_node_default_umask          Node 8    Applied     umask

logout_zsh_clear_screen           Node 40   Not Applied /etc/zlogout

logout_profile_clear_screen       Node 40   Not Applied /etc/profile

logout_csh_clear_screen           Node 40   Not Applied /etc/csh.logout

require_password_single_user_mode Node 40   Not Applied /etc/ttys

--More—(byte 2185)

These reports can be generated regardless of cluster hardening status.

OneFS security hardening comes pre-installed, but not activated by default, on a PowerScale cluster, and hardening cannot be uninstalled. Hardening is a licensed feature, and there are no changes to the licensing requirements or structure for OneFS 9.5 and later.

If a current license is not available, the following warning will be returned when attempting to activate security hardening on a cluster:

# isi hardening apply STIG

The HARDENING application is not currently installed. Please contact your Isilon account team for more information on evaluating and purchasing HARDENING.

Applying a hardening profile is one of multiple steps required in order to configure a STIG-compliant PowerScale cluster. In the next article in this series we’ll cover the configuration and activation of OneFS security hardening.

Another security enhancement that OneFS 9.5 brings to the table is the ability to configure 1GbE NIC ports dedicated to cluster management on the PowerScale F900, F600, F200 all-flash storage nodes and P100 and B100 accelerators. Since these platforms’ release, customers have been requesting the ability to activate the 1GbE NIC ports so that the node management activity and front end protocol traffic could be separated on physically distinct interfaces.

For background, the F600 and F900 have shipped with a quad port 1GbE rNDC (rack Converged Network Daughter Card) adapter since their introduction. However, these 1GbE ports were non-functional and unsupported in OneFS releases prior to 9.5. As such, the node management and front-end traffic was co-mingled on the front-end interface.

In OneFS 9.5 and later, 1GbE network ports are now supported on all of the PowerScale PowerEdge based platforms for the purposes of node management, and physically separate from the other network interfaces. Specifically, this enhancement applies to the F900, F600, F200 all-flash nodes, and P100 and B100 accelerators.

Under the hood, OneFS has been updated to recognize the 1GbE rNDC NIC ports as usable for a management interface. Note that the focus of this enhancement is on factory enablement and support for existing F600 customers that have that the unused 1GbE rNDC hardware. This functionality has also been back-ported to OneFS 9.4.0.3 and later RUPs. Since the introduction of this feature, there have been several requests raised about field upgrades, but that use case is separate and will be addressed in a later release via scripts, updates of node receipts, procedures, etc.

Architecturally, barring some device driver and accounting work, no substantial changes were required to the underlying OneFS or platform architecture in order to implement this feature. This means that, in addition to activating the rNDC, OneFS now supports the relocated front-end NIC in PCI slots 2 or 3 for the F200, B100, and P100.

OneFS 9.5 and later recognizes the 1GbE rNDC as usable for the management interface in the OneFS Wizard, in the same way it always has for the H-series and A-series chassis-based nodes.

All four ports in the 1GbE NIC are active, and, for the Broadcom board, the interfaces are initialized and reported as bge0, bge1, bge2, and bge3.

The ‘pciconf’ CLI utility can be used to determine whether the rNDC NIC is present in a node. If it is, a variety of identification and configuration details are displayed. For example, the following output from a Broadcom rNDC NIC in an F200 node:

# pciconf -lvV pci0:24:0:0

bge2@pci0:24:0:0: class=0x020000 card=0x1f5b1028 chip=0x165f14e4 rev=0x00 hdr=0x00

class       = network

subclass    = ethernet

VPD ident   = ‘Broadcom NetXtreme Gigabit Ethernet’

VPD ro PN   = ‘BCM95720’

VPD ro MN   = ‘1028’

VPD ro V0   = ‘FFV7.2.14’

VPD ro V1   = ‘DSV1028VPDR.VER1.0’

VPD ro V2   = ‘NPY2’

VPD ro V3   = ‘PMT1’

VPD ro V4   = ‘NMVBroadcom Corp’

VPD ro V5   = ‘DTINIC’

VPD ro V6   = ‘DCM1001008d452101000d45’

The ‘ifconfig’ CLI utility can be used to determine the specific IP/interface mapping on the Broadcom rNDC interface. For example:

# ifconfig bge0
TME-1: bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
TME-1:      ether 00:60:16:9e:X:X
TME-1:      inet 10.11.12.13 netmask 0xffffff00 broadcast 10.11.12.255 zone 1
TME-1:      inet 10.11.12.13 netmask 0xffffff00 broadcast 10.11.12.255 zone 0
TME-1:      media: Ethernet autoselect (1000baseT <full-duplex>)
TME-1:      status: active

In the above output, the first IP address of the management interface’s pool is bound to ‘bge0’, the first port on the Broadcom rNDC NIC.

The ‘isi network pools’ CLI command can be used to determine the corresponding interface. Within the system zone, the management interface is allocated an address from the configured IP range within its associated interface pool. For example:

# isi network pools list
ID                           SC Zone                       IP Ranges                   Allocation Method
---------------------------------------------------------------------------------------------------------
groupnet0.mgt.mgt          cluster_mgt_isln.com    10.11.12.13-10.11.12.20 static

# isi network pools view groupnet0.mgt.mgt | grep -i ifaces

Ifaces: 1:mgmt-1, 2:mgmt-1, 3:mgmt-1, 4:mgmt-1, 5:mgmt-1

Or from the WebUI, under Cluster management >Network configuration > External network:

Drilling down into the ’mgt’ pool details show the 1GbE management interfaces as the pool interface members:

Note that the 1GbE rNDC network ports are solely intended as cluster management interfaces. As such, they are not supported for use with regular front-end data traffic.

While the F900 and F600 nodes already ship with a four port 1GbE rNDC NIC installed. However, the F200, B100, and P100 platform configurations have been updated to also include a quad port 1GbE rNDC card, and these new configurations have been shipping by default since January 2023. This necessitated relocating the front end network’s 25GbE NIC (Mellanox CX4) to PCI slot 2 in the motherboard.  Additionally, the OneFS updates needed for this feature has also allowed the F200 platform to now be offered with a 100GbE option, too. The 100GbE option utilizes a Mellanox CX6 NIC in place of the CX4 in slot 2.

With this 1GbE management interface enhancement, the same quad-port rNDC card (typically the Broadcom 5720) that has been shipped in the F900 and F600 since their introduction, is now included in the F200, B100 and P100 nodes as well. All four 1GbE rNDC ports are enabled and active under OneFS 9.5 and later, too.

Node port ordering continues to follow the standard, increasing numerically from left to right. However, be aware that the port labels are not visible externally since they are obscured by the enclosure’s sheet metal.

The following back-of-chassis hardware images show the new placements of the NICs in the various F-series and accelerator platforms:

PowerScale F600

PowerScale F900

For both the F600 and F900, the NIC placement remains unchanged, since these nodes have always shipped with the 1GbE quad port in the rNDC slot since their launch.

PowerScale F200

The F200 sees its front-end NIC moved to slot 3, freeing up the rNDC slot for the quad-port 1GbE Broadcom 5720.

PowerScale B100

Since the B100 backup accelerator has a fibre-channel card in slot 2, it sees its front-end NIC moved to slot 3, freeing up the rNDC slot for the quad-port 1GbE Broadcom 5720.

PowerScale P100

Finally, the P100 accelerator sees its front-end NIC moved to slot 3, freeing up the rNDC slot for the quad-port 1GbE Broadcom 5720.

Note that, while there is currently no field hardware upgrade process for adding rNDC cards to legacy F200 nodes or B100 and P100 accelerators, this will likely be addressed in a future release.

As we’ve seen over the course of the last several articles, OneFS 9.5 delivers a wealth of security focused features. These span the realms of core file system, protocols, data services, platform, and peripherals. Among these security enhancements is the ability to manually or automatically disable a cluster’s USB ports from either the CLI, platform API, or via activation of a security hardening policy.

In support of this functionality, the basic USB port control architecture is as follows:

To facilitate this, OneFS 9.5 and subsequent releases see the addition of a new gconfig variable, ‘usb_ports_disabled’, in ‘security_config’, specifically to track the status of USB Ports on cluster. On receiving an admin request via either the CLI or platform API handler to disable the USB port, OneFS modifies the security config parameter in gconfig, as follows:

# isi_gconfig -t security_config | grep -i usb
usb_ports_disabled (bool) = true

Under the hood, the MCP (master control process) daemon watches for any changes to the ‘isi_security.gcfg’ security config file on the cluster. If the value for the ‘usb_ports_disabled’ variable in the ‘isi_security.gcfg’ file is updated, then MCP executes the ‘isi_config_usb’ utility to enact the desired change. Note that, since ‘isi_config_usb’ operates per-node but the MCP actions are global (executed cluster wide), isi_config_usb is invoked across each node via a python script to enable or disable the cluster’s USB Ports.

Be aware that the USB port control functionality is only supported PowerScale F900, F700, F200, H700/7000, and A300/3000 clusters running OneFS 9.5 or later.

Administratively, USB port control can be manually configured from either the CLI or platform API.

Note that there is no WebUI option at this time.

The CLI and platform API configuration options for USB port control in OneFS 9.5 and later are as follows

Action	CLI Syntax	Description
View	isi security settings view	Report the state of a cluster’s USB ports.
Enable	isi security settings modify –usb-ports-disabled=False	Activate a cluster’s USB ports.
Disable	isi security settings modify –usb-ports-disabled=True	Disable a cluster’s USB ports.

For example:

# isi security settings view | grep -i usb

      USB Ports Disabled: No

# isi security settings modify --usb-ports-disabled=True

# isi security settings view | grep -i usb

      USB Ports Disabled: Yes

Similarly, to re-enable a cluster’s USB ports:

# isi security settings modify --usb-ports-disabled=False

# isi security settings view | grep -i usb

      USB Ports Disabled: No

Note that a user account with the OneFS ISI_PRIV_CLUSTER RBAC privilege is required in order to configure USB port changes on a cluster.

In addition to the ‘isi security settings’ CLI command, there is also a node-local CLI utility:

# whereis isi_config_usb

isi_config_usb: /usr/bin/isi_hwtools/isi_config_usb

The general syntax of this utility is as follows:

 isi_config_usb [-h] [--nodes NODES] --mode {display,on,off}

As mentioned previously, ‘isi security settings’ acts globally on a cluster, using ‘isi_config_usb’ to effect its changes on each node.

Alternatively, cluster USB ports can also be enabled and disabled via the OneFS platform API with the following endpoints:

API	Method	Argument	Output
/16/security/settings	GET	No argument required.	JSON object for security settings with USB ports setting.
/16/security/settings	PUT	JSON object with boolean value for USB ports setting.	None or Error.

For example:

# curl -k -u <username>:<passwd> https://localhost:8080/platform/security/settings”

{
"settings" :
{
"fips_mode_enabled" : false,
"restricted_shell_enabled" : false,
"usb_ports_disabled" : true
}
}

In addition to manual configuration, the USB ports are automatically disabled if the STIG security hardening profile is applied to a cluster:

This is governed by the following section of XML code in the cluster hardening configuration XML file, located at /etc/isi_hardening/profiles/isi_hardening.xml:

<CONFIG_ITEM id ="isi_usb_ports" version = "1">
             <PapiOperation>
                          <DO>
                                       <URI>/security/settings</URI>
                                       <BODY>{"usb_ports_disabled": true}</BODY>
                                       <KEY>settings</KEY>
                          </DO>
                          <UNDO>
                                       <URI>/security/settings</URI>
                                       <BODY>{"usb_ports_disabled": false}</BODY>
                                       <KEY>settings</KEY>
                          </UNDO>
                          <ACTION_SCOPE>CLUSTER</ACTION_SCOPE>
                          <IGNORE>FALSE</IGNORE>
             </PapiOperation>
</CONFIG_ITEM>

The ‘isi_config_usb’ CLI utility can be used to display the USB port status on a subset of nodes. For example:

# isi_config_usb --nodes 1-10 --mode display

   Node   |  Current  |  Pending

-----------------------------------

   TME-9  |   UNSUP   | INFO: This platform is not supported to run this script.

   TME-8  |   UNSUP   | INFO: This platform is not supported to run this script.

   TME-1  |     On    |

   TME-3  |     On    |

   TME-2  |     On    |

  TME-10  |     On    |

   TME-7  |   AllOn   |

   TME-5  |   AllOn   |

   TME-6  |   AllOn   |

Unable to connect: TME-4

Note that, in addition to port status, the output will identify any nodes that do not support USB port control (nodes 8 and 9 above) or that are unreachable (node 4 above).

When investigating or troubleshooting issues with USB port control, the following log files are the first places to check:

Log file	Description
/var/log/isi_papi_d.log	Will log any requests to enable or disable USB ports.
/var/log/isi_config_usb.log	Logs activity from the isi_config_usb script execution.
/var/log/isi_mcp	Logs activity related to MCP actions on invoking the API.

The OneFS ‘isi_gather_info’ utility is the ubiquitous method for collecting and uploading a PowerScale cluster’s context and configuration to assist with the identification and resolution of bugs and issues. As such, it performs the following roles:

• Executes many commands, scripts, and utilities on a cluster, and saves their results.
• Collates, or gathers, all these files into a single ‘gzipped’ package.
• Optionally transmits this log gather package back to Dell via a choice of several transport methods.

By default, a log gather tarfile is written to the /ifs/data/Isilon_Support/pkg/ directory. It can also be uploaded to Dell via the following means:

Upload Mechanism	Description	TCP Port	OneFS Release Support
SupportAssist / ESRS	Uses Dell Secure Remote Support (SRS) for gather upload.	443/8443	Any
FTP	Use FTP to upload completed gather.	21	Any
FTPS	Use SSH-based encrypted FTPS to upload gather.	22	Default in OneFS 9.5 and later
HTTP	Use HTTP to upload gather.	80/443	Any

As indicated in the table above, OneFS 9.5 and later releases now leverage FTPS as the default option for FTP upload, thereby protecting the upload of cluster configuration and logs with an encrypted transmission session.

Under the hood, the log gather process comprises an eight phase workflow, with transmission comprising the penultimate ‘upload’ phase:

The details of each phase are as follows:

Phase	Description
1.       Setup	Reads from the arguments passed in, as well as any config files on disk, and sets up the config dictionary, which will be used throughout the rest of the codebase. Most of the code for this step is contained in isilon/lib/python/gather/igi_config/configuration.py. This is also the step where the program is most likely to exit, if some config arguments end up being invalid
2.       Run local	Executes all the cluster commands, which are run on the same node that is starting the gather. All these commands run in parallel (up to the current parallelism value). This is typically the second longest running phase.
3.       Run nodes	Executes the node commands across all of the cluster’s nodes. This runs on each node, and while these commands run in parallel (up to the current parallelism value), they do not run in parallel with the local step.
4.       Collect	Ensures all of the results end up on the overlord node (the node that started gather). If gather is using /ifs, it is very fast, but if it’s not, it needs to SCP all the node results to a single node.
5.       Generate Extra Files	Generates nodes_info and package_info.xml. These are two files that are present in every single gather, and tell us some important metadata about the cluster
6.       Packing	Packs (tars and gzips) all the results. This is typically the longest running phase, often by an order of magnitude
7.       Upload	Transports the tarfile package to its specified destination via SupportAssist, ESRS, FTPS, FTP, HTTP, etc. Depending on the geographic location, this phase might also be a lengthy duration.
8.       Cleanup	Cleanups any intermediary files that were created on cluster. This phase will run even if gather fails, or is interrupted.

Since the isi_gather_info tool is primarily intended for troubleshooting clusters with issues, it runs as root (or compadmin in compliance mode), as it needs to be able to execute under degraded conditions (eg. without GMP, during upgrade, and under cluster splits, etc). Given these atypical requirements, isi_gather_info is built as a stand-alone utility, rather than using the platform API for data collection.

While FTPS is the new default and recommend transport, the legacy plaintext FTP upload method is still available in OneFS 9.5 and beyond. As such, Dell’s log server, ftp.isilon.com, also supports both encrypted FTPS and plaintext FTP, so will not impact older release FTP log upload behavior.

This OneFS 9.5 FTPS security enhancement encompasses three primary areas where an FTPS option is now supported:

• ‘/usr/bin/isi_gather_info’ utility direct execution.
• Running via the ‘isi diagnostics gather’ CLI command set.
• A diagnostics gather through the OneFS WebUI.

For the ‘isi_gather_info’ script, two new options are added in OneFS 9.5:

New isi_gather_info Option	Description	Default Value
–ftp-insecure	Flag enables gather to use unencrypted FTP transfer.	False
–ftp-ssl-cert	Enables user to specify location of special SSL certificate file.	Empty spring. Not typically required.

Similarly, two new corresponding options for the ‘isi diagnostics’ CLI command:

New isi diagnostics Option	Description	Default Value
–ftp-upload-insecure	Flag enables gather to use unencrypted FTP transfer.	No
–ftp-upload-ssl-cert	Enables user to specify location of special SSL certificate file.	Empty spring. Not typically required.

Per the above, the following table shows command syntax usage examples for both FTPS and FTP uploads:

FTP Upload Type	Description	Example isi_gather_info Syntax	Example isi diagnostics Syntax
Secure upload (default)	Upload cluster logs to the Dell log server (ftp.isilon.com) via encrypted FTP (FTPS).	# isi_gather_info Or # isi_gather_info –ftp	# isi diagnostics gather start Or # isi diagnostics gather start –ftp-upload-insecure=no
Secure upload	Upload cluster logs to an alternative server via encrypted FTPS.	# isi_gather_info –ftp-host <FQDN> –ftp-ssl-cert <SSL_CERT_PATH>	# isi diagnostics gather start –ftp-upload-host=<FQDN> –ftp-ssl-cert= <SSL_CERT_PATH>
Unencrypted upload	Upload cluster logs to the Dell log server (ftp.isilon.com) via plain text FTP.	# isi_gather_info –ftp-insecure	# isi diagnostics gather start –ftp-upload-insecure=yes
Unencrypted upload	Upload cluster logs to an alternative server via plain text FTP.	# isi_gather_info –ftp-insecure –ftp-host <FQDN>	# isi diagnostics gather start –ftp-upload-host=<FQDN> –ftp-upload-insecure=yes

Note that OneFS 9.5 and later releases provide a warning if the cluster admin elects to continue using non-secure FTP as the for the isi_gather_info tool. Specifically, if the ‘–ftp-insecure’ option is configured, the following message is displayed, informing the user that plain text FTP upload is being used, and that the connection and data stream will not be encrypted.

# isi_gather_info --ftp-insecure

You are performing plain text FTP logs upload.

This feature is deprecated and will be removed

in a future release. Please consider the possibility

of using FTPS for logs upload. For further information,

please contact PowerScale support

...

In addition to the command line, log gathers can also be configured via the OneFS WebUI by navigating to Cluster management > Diagnostics > Gather settings:

The ‘Edit gather settings’ page in OneFS 9.5 and later has been updated to reflect FTPS as the default transport, plus the addition of radio buttons and text boxes to accommodate the new configuration options:

If plain text FTP upload is configured, healthcheck will display a warning that plain-text upload is used and is no longer a recommended option. For example:

For reference, the OneFS 9.5 and later ‘isi_gather_info’ CLI command syntax includes the following options:

Option	Description
–upload <boolean>	Enable gather upload.
–esrs <boolean>	Use ESRS for gather upload.
–noesrs	Do not attempt to upload via ESRS.
–supportassist	Attempt SupportAssist upload.
–nosupportassist	Do not attempt to upload via SupportAssist.
–gather-mode (incremental | full)	Type of gather: incremental, or full.
–http-insecure <boolean>	Enable insecure HTTP upload on completed gather.
–http -host <string>	HTTP Host to use for HTTP upload.
–http -path <string>	Path on HTTP server to use for HTTP upload.
–http -proxy <string>	Proxy server to use for HTTP upload.
–http -proxy-port <integer>	Proxy server port to use for HTTP upload.
–ftp <boolean>	Enable FTP upload on completed gather.
–noftp	Do not attempt FTP upload.
–set-ftp-password	Interactively specify alternate password for FTP.
–ftp -host <string>	FTP host to use for FTP upload.
–ftp -path <string>	Path on FTP server to use for FTP upload.
–ftp-port <string>	Specifies alternate FTP port for upload.
–ftp-proxy <string>	Proxy server to use for FTP upload.
–ftp -proxy-port <integer>	Proxy server port to use for FTP upload.
–ftp-mode <value>	Mode of FTP file transfer. Valid values are: both, active, passive
–ftp -user <string>	FTP user to use for FTP upload.
–ftp-pass <string>	Specify alternative password for FTP.
–ftp -ssl-cert <string>	Specifies the SSL certificate to use in FTPS connection.
–ftp-upload-insecure <boolean>	Whether to attempt a plain text FTP upload.
–ftp-upload-pass <string>	FTP user to use for FTP upload password.
–set-ftp-upload-pass	Specify the FTP upload password interactively.

Once a logfile gather arrives at Dell, it is automatically unpacked by a support process and analyzed using the ‘logviewer’ tool.

In the POSIX world, files typically possess three fundamental timestamps:

Timestamp	Alias	Description
Access	atime	Access timestamp of the last read.
Change	ctime	Status change timestamp of the last update to the file’s metadata.
Modify	mtime	Modification timestamp of the last write.

These timestamps can be easily viewed from a variety of file system tools and utilities. For example, in this case running ‘stat’ from the OneFS CLI:

# stat -x tstr

  File: "tstr"

  Size: 0            FileType: Regular File

  Mode: (0600/-rw-------)         Uid: (    0/    root)  Gid: (    0/   wheel)

Device: 18446744073709551611,18446744072690335895   Inode: 5103485107    Links: 1

Access: Mon Sep 11 23:12:47 2023

Modify: Mon Sep 11 23:12:47 2023

Change: Mon Sep 11 23:12:47 2023

A typical instance of a change, or  ‘ctime’, timestamp update occurs when a file’s access permissions are altered. Since modifying the permissions doesn’t physically open the file (ie. access the file’s data), its ‘atime’ field is not updated. Similarly, since no modification is made to the file’s contents the ‘mtime’ also remains unchanged. However, the file’s metadata has been changed, and the ctime field is used to record this event. As such, the ‘ctime’ stamp allows a workflow such as a backup application to know to make a fresh copy of the file, including its updated permission values. Similarly, a file rename is another operation that modifies its ‘ctime’ entry without affecting the other timestamps.

Certain other file systems also include a fourth timestamp: namely the ‘birthtime’ of when the file was created. Birthtime (by definition) should never change. It’s also an attribute which organizations and their storage administrators may or may not care about.

Within the Windows file system realm, this ‘birthtime’ timestamp, is affectionally known as ‘create date’. This create date of a file is essentially the date and time when its inode is ‘born’. Note that this is not a recognized POSIX attribute, like ctime or mtime, rather it is something that was introduced as part of Windows compatibility requirements. And, because it’s a birthtime, linking operations do not necessarily affect it unless a new inode is not created.

As shown below, this create, or birth, date can differ from a file’s modified or accessed dates because the creation date is when that file’s inode version originated. So, for instance, if a file is copied, the new file’s create date will be set to the current time since it has a new inode. This can be seen in the following example where a file is copied from a flash drive mounted on a Windows client’s file system under drive ‘E:’, to a cluster’s SMB share mounted at drive ‘Z:’.

The ‘Date created’ date above is ahead in time of both the ‘accessed’ and ‘modified’, because the latter two were merely inherited from the source file, whereas the create date was set when the copy was made.

The corresponding ‘date’, ‘stat’, and ‘isi get’ CLI output from the cluster confirms this:

In before OneFS 9.5, when a file is replicated, its create date is timestamped when that file was copied from the source cluster. This means when the replication job ran, or, more specifically, when the individual job worker thread got around to processing that specific file.

By way of contrast, OneFS 9.5 now ensures that SyncIQ fully replicates the full array of metadata, preserving all values, including that of the birth time / create date.

The primary consideration for the new create date functionality is that it requires both source and target clusters in a replication set to be running OneFS 9.5 or later.

If either the source or the target is running pre-9.5 code, this time field retains its old behavior of being set to the time of replication (actual file creation) rather than the correct value associated with the source file.

In OneFS 9.5, create date timestamping works exactly the same way as SyncIQ replication of other metadata (such as ‘mtime’, etc), occurring automatically as part of every file replication. Plus, no additional configuration is necessary beyond upgrading both clusters to OneFS 9.5.

One other significant thing to note about this feature is that SyncIQ is changelist-based, using OneFS snapshots under the hood for its checkpointing and delta comparisons.. This means that, if a replication relationship has been configured prior to OneFS 9.5 upgrade, the source cluster will have valid birthtime data, but the target’s birthtime data will reflect the local creation time of the files it’s copied.

Note that, upon upgrading both sides to OneFS 9.5 and running a SyncIQ job, nothing will change. This is  because SyncIQ will perform its snapshot comparison, determine that no changes were made to the dataset, and so will not perform any replication work. However, if a source file is ‘touched’ so that it’s mtime is changed (or any other action performed that will cause a copy-on-write, or CoW) that will cause the file to show up in the snapshot diff and therefore be replicated. As part of replicating that file, the correct birth time will be written on the target.

Note that a full replication (re)sync does not get triggered as a result of upgrading a replication cluster pair to OneFS 9.5 or later and thereby enabling this functionality. Instead, any create date timestamp resolution happens opportunistically and in the background as files gets touched or modified – and thereby replicated. Be aware that ‘touching’ a file does change its modification time, in addition to updating the create date, which may be undesirable.

In the previous article in this series we looked at the architecture and operation of OneFS configuration auditing. Now, we’ll turn our attention to its management, event viewing, and troubleshooting.

The CLI command set for configuring ‘isi audit’ is split between two functional areas:

Area	Detail	Syntax
Events	Specifies which specific events get logged, across three categories: ·         Audit Failure ·         Audit Success ·         Syslog Audit Events	isi audit settings …
Global	Configuration of global audit parameters, including topics, zones, CEE, syslog, puring, retention, etc.	isi audit settings global …

The ‘view’ argument for each command returns the following output:

1. Events:

# isi audit settings view

            Audit Failure: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

            Audit Success: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

      Syslog Audit Events: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

Syslog Forwarding Enabled: No

2. Global:

# isi audit settings global view

     Protocol Auditing Enabled: Yes

                 Audited Zones: -

               CEE Server URIs: -

                      Hostname:

       Config Auditing Enabled: Yes

         Config Syslog Enabled: No

         Config Syslog Servers: -

     Config Syslog TLS Enabled: No

  Config Syslog Certificate ID:

       Protocol Syslog Servers: -

   Protocol Syslog TLS Enabled: No

Protocol Syslog Certificate ID:

         System Syslog Enabled: No

         System Syslog Servers: -

     System Syslog TLS Enabled: No

  System Syslog Certificate ID:

          Auto Purging Enabled: No

              Retention Period: 180

       System Auditing Enabled: No

While configuration auditing is disabled on OneFS by default, the following CLI syntax can be used enable and verify config auditing across the cluster:

# isi audit settings global modify --config-auditing-enabled 1

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Similarly, to enable configuration change audit redirection to syslog:

# isi audit settings global modify --config-auditing-enabled true

# isi audit settings global modify --config-syslog-enabled true

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Or to disable redirection to syslog:

# isi audit settings global modify --config-syslog-enabled false

# isi audit settings global modify --config-auditing-enabled false

Up to six CEE servers per cluster can be configured. For example:

#isi audit settings global modify --add-cee-server-uris='<URL>’ For example: #isi audit settings global modify --add-cee-server-uris='http://cee1.isilon.com:12228/cee'

Auditing can be constrained by access zone, too:

# isi audit settings modify --add-audited-zones=audit_az1

Note that, when auditing is enabled, the system zone is included by default. However, it can be excluded, if desired:

# isi audit setting modify --remove-audited-zones=System

Access zone’s audit parameters can also be configured via the ‘isi zones’ CLI command set. For example:

# isi zone zones create --all-auth-providers=true --audit-failure=all --audit-success=all --path=/ifs/data --name=audit_az1

Granular audit event type configuration can be specified, if desired, to narrow the scope and reduce the amount of audit logging, etc.

For example, the following command syntax constrains auditing to read and logon failures and successful writes and deletes under path /ifs/data in the audit_az1 access zone:

# isi zone zones create --all-auth-providers=true --audit-failure=read,logon --audit-success=write,delete --path=/ifs/data --name=audit_az1

In addition to the CLI, the OneFS platform API can also be used to configure and manage auditing. For example, to enable configuration auditing on a cluster:

PUT /platform/1/audit/settings

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

        {

               'config_auditing_enabled': True

        }

The following ‘204’ HTTP response code from the cluster indicates that the request was successful, and that configuration auditing is now enabled on the cluster. No message body is returned for this request.

204 No Content

Content-type: text/plain,

Allow: 'GET, PUT, HEAD'

Similarly, to modify the config audit topic’s maximum cached messages threshold to a value of ‘1000’ via the API:

PUT /1/audit/topics/config

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

    {

        "max_cached_messages": 1000

    }

Again, no message body is returned from OneFS for this request.

204 No Content

Content-type: text/plain,

Allow: 'GET, PUT, HEAD'

Note that, in the unlikely event that a cluster experiences an outage during which it loses quorum, auditing will be suspended until it is regained. Events similar to the following will be written to the /var/log/audit_d.log file:

940b5c700]: Lost quorum! Audit logging will be disabled until /ifs is writeable again.

2023-08-28T15:37:32.132780+00:00 <1.6> TME-1(id1) isi_audit_d[6495]: [0x345940b5c700]: Regained quorum. Logging resuming.

When it comes to reading audit events on the cluster, OneFS natively provides the handy ‘isi_audit_viewer’ utility. For example, the following audit viewer output shows the events logged when the cluster admin added the ‘/ifs/tmp’ path to the SmartDedupe configuration, and created a new user named ‘test’1’:

# isi_audit_viewer

[0: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076315499,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/1/dedupe/settings","method":"PUT","args":{}

,"body":{"paths":["/ifs/tmp"]}

}}

[1: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076391422,"payload":{"status":204,"statusmsg":"No Content","body":{}}}

[2: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223446993,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/18/auth/users","method":"POST","args":{}

,"body":{"name":"test1"}

}}

[3: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223507797,"payload":{"status":201,"statusmsg":"Created","body":{"id":"SID:S-1-5-21-593535466-4266055735-3901207217-1000"}

}}

The audit log entries, such as those above, typically comprise the following components:

Order	Component	Detail
1	Timestamp	Timestamp in human readable form
2	ID	Unique entry ID
3	Timestamp	Timestamp in UNIX epoch time
4	Node	Node number
5	User tokens	1.       The user tokens of the Roles and rights of user executing the command ·         User persona (Unix/Windows) ·         Primary group persona (Unix/Windows) ·         Supplemental group personas (Unix/Windows) ·         RBAC privileges of the user executing the command
6	Interface	Interface used to generate the command: ·         10 = pAPI / WebUI ·         16 = Console CLI ·         17 = SSH CLI
7	Zone	Access zone that the command was executed against
8	Client IP	Where the user connected from
9	Local node	Local node address where the command was executed
10	Command	Command syntax
11	Arguments	Command arguments
12	Body	Command body

The ‘isi_audit_viewer’ utility automatically reads the ‘config’ log topic by default, but can also be used read the ‘protocol’ log topic too. Its CLI command syntax is as follows:

# isi_audit_viewer -h

Usage: isi_audit_viewer [ -n <nodeid> | -t <topic> | -s <starttime>|

         -e <endtime> | -v ]

         -n <nodeid> : Specify node id to browse (default: local node)

         -t <topic>  : Choose topic to browse.

            Topics are "config" and "protocol" (default: "config")

         -s <start>  : Browse audit logs starting at <starttime>

         -e <end>    : Browse audit logs ending at <endtime>

         -v verbose  : Prints out start / end time range before printing records

Note that, on large clusters where there is heavy (ie. 100,000’s) of audit writes, when running the isi_audit_viewer utility across the cluster with ‘isi_for_array’, it can potentially lead to memory starvation and other issues – especially if outputting to a directory under /ifs. As such, consider directing the output to a non-IFS location such as /var/temp. Also, the isi_audit_viewer ‘-s’ (start time) and ‘-e’ (end time) flags can be used to limit a search (ie. for  1-5 minutes), helping reduce the size of data.

In addition to reading audit events, the view is also a useful tool to assist with troubleshoot any auditing issues. Additionally, any errors that are encountered while processing audit events, and when delivering them to an external CEE server, are written to the log file ‘/var/log/isi_audit_cee.log’. Additionally, the protocol specific logs will contain any issues the audit filter has collecting while auditing events.

OneFS auditing can detect potential sources of data loss, fraud, inappropriate entitlements, access attempts that should not occur, and a range of other anomalies that are indicators of risk. This can be especially useful when the audit associates data access with specific user identities.

In the interests of data security, OneFS provides ‘chain of custody’ auditing by logging specific activity on the cluster. This includes OneFS configuration changes plus NFS, SMB, and HDFS client protocol activity, which are required for organizational IT security compliance, as mandated by regulatory bodies like HIPAA, SOX, FISMA, MPAA, etc.

OneFS auditing uses Dell’s Common Event Enabler (CEE) to provide compatibility with external audit applications. A cluster can write audit events across up to five CEE servers per node in a parallel, load-balanced configuration. This allows OneFS to deliver an end to end, enterprise grade audit solution which efficiently integrates with third party solutions like Varonis DatAdvantage.

The following diagram outlines the basic architecture of OneFS audit:

Both system configuration changes, as well as protocol activity, can be easily audited on a PowerScale cluster. However, the protocol path is greyed out above, since it is outside the focus of this article. More information on OneFS protocol auditing can be found here.

As illustrated above, the OneFS audit framework is centered around three main services.

Service	Description
isi_audit_cee	Service allowing OneFS to support third-party auditing applications. The main method of accessing protocol audit data from OneFS is through a third-party auditing application.
isi_audit_d	Responsible for per-node audit queues and managing the data store for those queues. It provides a protocol on which clients may produce event payloads within a given context. It establishes a Unix domain socket for queue producers and handles writing and rotation of log files in /ifs/.ifsvar/audit/logs/node###/{config,protocol}/*.
isi_audit_syslog	Daemon providing forwarding of audit config and protocol events to syslog.

The basic configuration auditing workflow sees a cluster config change request come in via either the OneFS CLI, WebUI or platform API. The API handler infrastructure passes this request to the isi_audit_d service, which intercepts it as a client thread and adds it to the audit queue. It is then processed and passed via a backend thread and written to the audit log files (IFS) as appropriate.

If audit syslog forwarding has been configured, IFS also passes the event to the isi_audit_syslog daemon, where a supervisor process instructs a writer thread to send it to the syslog, which in turn updates its pertinent /var/log/ logfiles.

Similarly, if Common Event Enabler (CEE) forwarding has been enabled, IFS will also pass the request to the isi_audit_cee service, where a delivery worker threads will intercept it and send the event to the CEE server pool. The isi_audit_cee heartbeat task makes CEE servers available for audit event delivery. Only after a CEE server has received a successful heartbeat will audit events be delivered to it. Every ten seconds the heartbeat task wakes up and sends each CEE server in the configuration a heartbeat. While CEE servers are available and events are in memory an attempt will be made to deliver these. Shutdown will only save audit log position if all the events are delivered to CEE since audit should not lose events. It isn’t critical that all events are delivered at shutdown since any unsaved events can be resent to CEE on the next start of isi_audit_cee since CEE handles duplicates.

Within OneFS, all audit data is organized by topic and is securely stored in the file system.

# isi audit topics list

Name     Max Cached Messages

-----------------------------

protocol 2048

config   1024

-----------------------------

Total: 2

Auditing can detect a variety of potential sources of data loss. These include unauthorized access attempts, inappropriate entitlements, plus a bevy of other fraudulent activities that plague organizations across the gamut of industries. Enterprises are increasingly tasked required to comply with stringent regulatory mandates developed to protect against these sources of data theft and loss.

OneFS system configuration auditing is designed to track and record all configuration events that are handled by the API through the command-line interface (CLI).

# isi audit topics view config

               Name: config

Max Cached Messages: 1024

Once enabled, system configuration auditing requires no additional configuration, and auditing events are automatically stored in the config audit topic directories. Audit access and management is governed by the ‘ISI_PRIV_AUDIT’ RBAC privilege, and OneFS provides a default ‘AuditAdmin’ role for this purpose.

Audit events are stored in a binary file under /ifs/.ifsvar/audit/logs. The logs automatically roll over to a new file after the size reaches 1 GB. The audit logs are consumable by auditing applications that support the Dell Common Event Enabler (CEE).

OneFS audit topics and settings can easily be viewed and modified. For example, to increase the configuration auditing maximum cached messages threshold to 2048 from the CLI:

# isi audit topics modify config --max-cached-messages 2048

# isi audit topics view config

               Name: config

Max Cached Messages: 2048

Audit configuration can also be modified or viewed per access zone and/or topic.

Operation	CLI Syntax	Method and URI
Get audit settings	isi audit settings  view	GET <cluster-ip:port>/platform/3/audit/settings
Modify audit settings	isi audit settings modify …	PUT <cluster-ip:port>/platform/3/audit/settings
View JSON schema for this resource, including query parameters and object properties info.		GET <cluster-ip:port>/platform/3/audit/settings?describe
View JSON schema for this resource, including query parameters and object properties info.		GET <cluster-ip:port>/platform/1/audit/topics?describe

Configuration auditing can be enabled on a cluster from either the CLI or platform API. The current global audit configuration can be viewed as follows:

# isi audit settings global view

     Protocol Auditing Enabled: No

                 Audited Zones: -

               CEE Server URIs: -

                      Hostname:

       Config Auditing Enabled: No

         Config Syslog Enabled: No

         Config Syslog Servers: -

     Config Syslog TLS Enabled: No

  Config Syslog Certificate ID:

       Protocol Syslog Servers: -

   Protocol Syslog TLS Enabled: No

Protocol Syslog Certificate ID:

         System Syslog Enabled: No

         System Syslog Servers: -

     System Syslog TLS Enabled: No

  System Syslog Certificate ID:

          Auto Purging Enabled: No

              Retention Period: 180

       System Auditing Enabled: No

In this case, configuration auditing is disabled – its default setting. The following CLI syntax will enable (and verify) configuration auditing across the cluster:

# isi audit settings global modify --config-auditing-enabled 1

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

In the next article, we’ll look at the config audit management, event viewing, and troubleshooting.