OneFS System Configuration Auditing – Part 2

In the previous article in this series we looked at the architecture and operation of OneFS configuration auditing. Now, we’ll turn our attention to its management, event viewing, and troubleshooting.

The CLI command set for configuring ‘isi audit’ is split between two functional areas:

Area Detail Syntax
Events Specifies which specific events get logged, across three categories:

·         Audit Failure

·         Audit Success

·         Syslog Audit Events

 isi audit settings …
Global Configuration of global audit parameters, including topics, zones, CEE, syslog, puring, retention, etc. isi audit settings global …

The ‘view’ argument for each command returns the following output:

  1. Events:
# isi audit settings view

            Audit Failure: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

            Audit Success: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

      Syslog Audit Events: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

Syslog Forwarding Enabled: No
  1. Global:
# isi audit settings global view

     Protocol Auditing Enabled: Yes

                 Audited Zones: -

               CEE Server URIs: -

                      Hostname:

       Config Auditing Enabled: Yes

         Config Syslog Enabled: No

         Config Syslog Servers: -

     Config Syslog TLS Enabled: No

  Config Syslog Certificate ID:

       Protocol Syslog Servers: -

   Protocol Syslog TLS Enabled: No

Protocol Syslog Certificate ID:

         System Syslog Enabled: No

         System Syslog Servers: -

     System Syslog TLS Enabled: No

  System Syslog Certificate ID:

          Auto Purging Enabled: No

              Retention Period: 180

       System Auditing Enabled: No

While configuration auditing is disabled on OneFS by default, the following CLI syntax can be used enable and verify config auditing across the cluster:

# isi audit settings global modify --config-auditing-enabled 1

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Similarly, to enable configuration change audit redirection to syslog:

# isi audit settings global modify --config-auditing-enabled true

# isi audit settings global modify --config-syslog-enabled true

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Or to disable redirection to syslog:

# isi audit settings global modify --config-syslog-enabled false

# isi audit settings global modify --config-auditing-enabled false

Up to six CEE servers per cluster can be configured. For example:

#isi audit settings global modify --add-cee-server-uris='<URL>’ For example: #isi audit settings global modify --add-cee-server-uris='http://cee1.isilon.com:12228/cee'

Auditing can be constrained by access zone, too:

# isi audit settings modify --add-audited-zones=audit_az1

Note that, when auditing is enabled, the system zone is included by default. However, it can be excluded, if desired:

# isi audit setting modify --remove-audited-zones=System

Access zone’s audit parameters can also be configured via the ‘isi zones’ CLI command set. For example:

# isi zone zones create --all-auth-providers=true --audit-failure=all --audit-success=all --path=/ifs/data --name=audit_az1

Granular audit event type configuration can be specified, if desired, to narrow the scope and reduce the amount of audit logging, etc.

For example, the following command syntax constrains auditing to read and logon failures and successful writes and deletes under path /ifs/data in the audit_az1 access zone:

# isi zone zones create --all-auth-providers=true --audit-failure=read,logon --audit-success=write,delete --path=/ifs/data --name=audit_az1

In addition to the CLI, the OneFS platform API can also be used to configure and manage auditing. For example, to enable configuration auditing on a cluster:

PUT /platform/1/audit/settings

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

        {

               'config_auditing_enabled': True

        }

The following ‘204’ HTTP response code from the cluster indicates that the request was successful, and that configuration auditing is now enabled on the cluster. No message body is returned for this request.

204 No Content

Content-type: text/plain,

Allow: 'GET, PUT, HEAD'

Similarly, to modify the config audit topic’s maximum cached messages threshold to a value of ‘1000’ via the API:

PUT /1/audit/topics/config

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

    {

        "max_cached_messages": 1000

    }

Again, no message body is returned from OneFS for this request.

204 No Content

Content-type: text/plain,

Allow: 'GET, PUT, HEAD'

Note that, in the unlikely event that a cluster experiences an outage during which it loses quorum, auditing will be suspended until it is regained. Events similar to the following will be written to the /var/log/audit_d.log file:

940b5c700]: Lost quorum! Audit logging will be disabled until /ifs is writeable again.

2023-08-28T15:37:32.132780+00:00 <1.6> TME-1(id1) isi_audit_d[6495]: [0x345940b5c700]: Regained quorum. Logging resuming.

When it comes to reading audit events on the cluster, OneFS natively provides the handy ‘isi_audit_viewer’ utility. For example, the following audit viewer output shows the events logged when the cluster admin added the ‘/ifs/tmp’ path to the SmartDedupe configuration, and created a new user named ‘test’1’:

# isi_audit_viewer

[0: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076315499,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/1/dedupe/settings","method":"PUT","args":{}

,"body":{"paths":["/ifs/tmp"]}

}}

[1: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076391422,"payload":{"status":204,"statusmsg":"No Content","body":{}}}

[2: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223446993,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/18/auth/users","method":"POST","args":{}

,"body":{"name":"test1"}

}}

[3: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223507797,"payload":{"status":201,"statusmsg":"Created","body":{"id":"SID:S-1-5-21-593535466-4266055735-3901207217-1000"}

}}

The audit log entries, such as those above, typically comprise the following components:

Order Component Detail
1 Timestamp Timestamp in human readable form
2 ID Unique entry ID
3 Timestamp Timestamp in UNIX epoch time
4 Node Node number
5 User tokens 1.       The user tokens of the Roles and rights of user executing the command

·         User persona (Unix/Windows)

·         Primary group persona (Unix/Windows)

·         Supplemental group personas (Unix/Windows)

·         RBAC privileges of the user executing the command
6 Interface Interface used to generate the command:

·         10 = pAPI / WebUI

·         16 = Console CLI

·         17 = SSH CLI
7 Zone Access zone that the command was executed against
8 Client IP Where the user connected from
9 Local node Local node address where the command was executed
10 Command Command syntax
11 Arguments Command arguments
12 Body Command body

The ‘isi_audit_viewer’ utility automatically reads the ‘config’ log topic by default, but can also be used read the ‘protocol’ log topic too. Its CLI command syntax is as follows:

# isi_audit_viewer -h

Usage: isi_audit_viewer [ -n <nodeid> | -t <topic> | -s <starttime>|

         -e <endtime> | -v ]

         -n <nodeid> : Specify node id to browse (default: local node)

         -t <topic>  : Choose topic to browse.

            Topics are "config" and "protocol" (default: "config")

         -s <start>  : Browse audit logs starting at <starttime>

         -e <end>    : Browse audit logs ending at <endtime>

         -v verbose  : Prints out start / end time range before printing records

Note that, on large clusters where there is heavy (ie. 100,000’s) of audit writes, when running the isi_audit_viewer utility across the cluster with ‘isi_for_array’, it can potentially lead to memory starvation and other issues – especially if outputting to a directory under /ifs. As such, consider directing the output to a non-IFS location such as /var/temp. Also, the isi_audit_viewer ‘-s’ (start time) and ‘-e’ (end time) flags can be used to limit a search (ie. for  1-5 minutes), helping reduce the size of data.

In addition to reading audit events, the view is also a useful tool to assist with troubleshoot any auditing issues. Additionally, any errors that are encountered while processing audit events, and when delivering them to an external CEE server, are written to the log file ‘/var/log/isi_audit_cee.log’. Additionally, the protocol specific logs will contain any issues the audit filter has collecting while auditing events.

Leave a Reply

Your email address will not be published. Required fields are marked *