Month: September 2023

The OneFS ‘isi_gather_info’ utility is the ubiquitous method for collecting and uploading a PowerScale cluster’s context and configuration to assist with the identification and resolution of bugs and issues. As such, it performs the following roles:

• Executes many commands, scripts, and utilities on a cluster, and saves their results.
• Collates, or gathers, all these files into a single ‘gzipped’ package.
• Optionally transmits this log gather package back to Dell via a choice of several transport methods.

By default, a log gather tarfile is written to the /ifs/data/Isilon_Support/pkg/ directory. It can also be uploaded to Dell via the following means:

Upload Mechanism	Description	TCP Port	OneFS Release Support
SupportAssist / ESRS	Uses Dell Secure Remote Support (SRS) for gather upload.	443/8443	Any
FTP	Use FTP to upload completed gather.	21	Any
FTPS	Use SSH-based encrypted FTPS to upload gather.	22	Default in OneFS 9.5 and later
HTTP	Use HTTP to upload gather.	80/443	Any

As indicated in the table above, OneFS 9.5 and later releases now leverage FTPS as the default option for FTP upload, thereby protecting the upload of cluster configuration and logs with an encrypted transmission session.

Under the hood, the log gather process comprises an eight phase workflow, with transmission comprising the penultimate ‘upload’ phase:

The details of each phase are as follows:

Phase	Description
1.       Setup	Reads from the arguments passed in, as well as any config files on disk, and sets up the config dictionary, which will be used throughout the rest of the codebase. Most of the code for this step is contained in isilon/lib/python/gather/igi_config/configuration.py. This is also the step where the program is most likely to exit, if some config arguments end up being invalid
2.       Run local	Executes all the cluster commands, which are run on the same node that is starting the gather. All these commands run in parallel (up to the current parallelism value). This is typically the second longest running phase.
3.       Run nodes	Executes the node commands across all of the cluster’s nodes. This runs on each node, and while these commands run in parallel (up to the current parallelism value), they do not run in parallel with the local step.
4.       Collect	Ensures all of the results end up on the overlord node (the node that started gather). If gather is using /ifs, it is very fast, but if it’s not, it needs to SCP all the node results to a single node.
5.       Generate Extra Files	Generates nodes_info and package_info.xml. These are two files that are present in every single gather, and tell us some important metadata about the cluster
6.       Packing	Packs (tars and gzips) all the results. This is typically the longest running phase, often by an order of magnitude
7.       Upload	Transports the tarfile package to its specified destination via SupportAssist, ESRS, FTPS, FTP, HTTP, etc. Depending on the geographic location, this phase might also be a lengthy duration.
8.       Cleanup	Cleanups any intermediary files that were created on cluster. This phase will run even if gather fails, or is interrupted.

Since the isi_gather_info tool is primarily intended for troubleshooting clusters with issues, it runs as root (or compadmin in compliance mode), as it needs to be able to execute under degraded conditions (eg. without GMP, during upgrade, and under cluster splits, etc). Given these atypical requirements, isi_gather_info is built as a stand-alone utility, rather than using the platform API for data collection.

While FTPS is the new default and recommend transport, the legacy plaintext FTP upload method is still available in OneFS 9.5 and beyond. As such, Dell’s log server, ftp.isilon.com, also supports both encrypted FTPS and plaintext FTP, so will not impact older release FTP log upload behavior.

This OneFS 9.5 FTPS security enhancement encompasses three primary areas where an FTPS option is now supported:

• ‘/usr/bin/isi_gather_info’ utility direct execution.
• Running via the ‘isi diagnostics gather’ CLI command set.
• A diagnostics gather through the OneFS WebUI.

For the ‘isi_gather_info’ script, two new options are added in OneFS 9.5:

New isi_gather_info Option	Description	Default Value
–ftp-insecure	Flag enables gather to use unencrypted FTP transfer.	False
–ftp-ssl-cert	Enables user to specify location of special SSL certificate file.	Empty spring. Not typically required.

Similarly, two new corresponding options for the ‘isi diagnostics’ CLI command:

New isi diagnostics Option	Description	Default Value
–ftp-upload-insecure	Flag enables gather to use unencrypted FTP transfer.	No
–ftp-upload-ssl-cert	Enables user to specify location of special SSL certificate file.	Empty spring. Not typically required.

Per the above, the following table shows command syntax usage examples for both FTPS and FTP uploads:

FTP Upload Type	Description	Example isi_gather_info Syntax	Example isi diagnostics Syntax
Secure upload (default)	Upload cluster logs to the Dell log server (ftp.isilon.com) via encrypted FTP (FTPS).	# isi_gather_info Or # isi_gather_info –ftp	# isi diagnostics gather start Or # isi diagnostics gather start –ftp-upload-insecure=no
Secure upload	Upload cluster logs to an alternative server via encrypted FTPS.	# isi_gather_info –ftp-host <FQDN> –ftp-ssl-cert <SSL_CERT_PATH>	# isi diagnostics gather start –ftp-upload-host=<FQDN> –ftp-ssl-cert= <SSL_CERT_PATH>
Unencrypted upload	Upload cluster logs to the Dell log server (ftp.isilon.com) via plain text FTP.	# isi_gather_info –ftp-insecure	# isi diagnostics gather start –ftp-upload-insecure=yes
Unencrypted upload	Upload cluster logs to an alternative server via plain text FTP.	# isi_gather_info –ftp-insecure –ftp-host <FQDN>	# isi diagnostics gather start –ftp-upload-host=<FQDN> –ftp-upload-insecure=yes

Note that OneFS 9.5 and later releases provide a warning if the cluster admin elects to continue using non-secure FTP as the for the isi_gather_info tool. Specifically, if the ‘–ftp-insecure’ option is configured, the following message is displayed, informing the user that plain text FTP upload is being used, and that the connection and data stream will not be encrypted.

# isi_gather_info --ftp-insecure

You are performing plain text FTP logs upload.

This feature is deprecated and will be removed

in a future release. Please consider the possibility

of using FTPS for logs upload. For further information,

please contact PowerScale support

...

In addition to the command line, log gathers can also be configured via the OneFS WebUI by navigating to Cluster management > Diagnostics > Gather settings:

The ‘Edit gather settings’ page in OneFS 9.5 and later has been updated to reflect FTPS as the default transport, plus the addition of radio buttons and text boxes to accommodate the new configuration options:

If plain text FTP upload is configured, healthcheck will display a warning that plain-text upload is used and is no longer a recommended option. For example:

For reference, the OneFS 9.5 and later ‘isi_gather_info’ CLI command syntax includes the following options:

Option	Description
–upload <boolean>	Enable gather upload.
–esrs <boolean>	Use ESRS for gather upload.
–noesrs	Do not attempt to upload via ESRS.
–supportassist	Attempt SupportAssist upload.
–nosupportassist	Do not attempt to upload via SupportAssist.
–gather-mode (incremental | full)	Type of gather: incremental, or full.
–http-insecure <boolean>	Enable insecure HTTP upload on completed gather.
–http -host <string>	HTTP Host to use for HTTP upload.
–http -path <string>	Path on HTTP server to use for HTTP upload.
–http -proxy <string>	Proxy server to use for HTTP upload.
–http -proxy-port <integer>	Proxy server port to use for HTTP upload.
–ftp <boolean>	Enable FTP upload on completed gather.
–noftp	Do not attempt FTP upload.
–set-ftp-password	Interactively specify alternate password for FTP.
–ftp -host <string>	FTP host to use for FTP upload.
–ftp -path <string>	Path on FTP server to use for FTP upload.
–ftp-port <string>	Specifies alternate FTP port for upload.
–ftp-proxy <string>	Proxy server to use for FTP upload.
–ftp -proxy-port <integer>	Proxy server port to use for FTP upload.
–ftp-mode <value>	Mode of FTP file transfer. Valid values are: both, active, passive
–ftp -user <string>	FTP user to use for FTP upload.
–ftp-pass <string>	Specify alternative password for FTP.
–ftp -ssl-cert <string>	Specifies the SSL certificate to use in FTPS connection.
–ftp-upload-insecure <boolean>	Whether to attempt a plain text FTP upload.
–ftp-upload-pass <string>	FTP user to use for FTP upload password.
–set-ftp-upload-pass	Specify the FTP upload password interactively.

Once a logfile gather arrives at Dell, it is automatically unpacked by a support process and analyzed using the ‘logviewer’ tool.

In the POSIX world, files typically possess three fundamental timestamps:

Timestamp	Alias	Description
Access	atime	Access timestamp of the last read.
Change	ctime	Status change timestamp of the last update to the file’s metadata.
Modify	mtime	Modification timestamp of the last write.

These timestamps can be easily viewed from a variety of file system tools and utilities. For example, in this case running ‘stat’ from the OneFS CLI:

# stat -x tstr

  File: "tstr"

  Size: 0            FileType: Regular File

  Mode: (0600/-rw-------)         Uid: (    0/    root)  Gid: (    0/   wheel)

Device: 18446744073709551611,18446744072690335895   Inode: 5103485107    Links: 1

Access: Mon Sep 11 23:12:47 2023

Modify: Mon Sep 11 23:12:47 2023

Change: Mon Sep 11 23:12:47 2023

A typical instance of a change, or  ‘ctime’, timestamp update occurs when a file’s access permissions are altered. Since modifying the permissions doesn’t physically open the file (ie. access the file’s data), its ‘atime’ field is not updated. Similarly, since no modification is made to the file’s contents the ‘mtime’ also remains unchanged. However, the file’s metadata has been changed, and the ctime field is used to record this event. As such, the ‘ctime’ stamp allows a workflow such as a backup application to know to make a fresh copy of the file, including its updated permission values. Similarly, a file rename is another operation that modifies its ‘ctime’ entry without affecting the other timestamps.

Certain other file systems also include a fourth timestamp: namely the ‘birthtime’ of when the file was created. Birthtime (by definition) should never change. It’s also an attribute which organizations and their storage administrators may or may not care about.

Within the Windows file system realm, this ‘birthtime’ timestamp, is affectionally known as ‘create date’. This create date of a file is essentially the date and time when its inode is ‘born’. Note that this is not a recognized POSIX attribute, like ctime or mtime, rather it is something that was introduced as part of Windows compatibility requirements. And, because it’s a birthtime, linking operations do not necessarily affect it unless a new inode is not created.

As shown below, this create, or birth, date can differ from a file’s modified or accessed dates because the creation date is when that file’s inode version originated. So, for instance, if a file is copied, the new file’s create date will be set to the current time since it has a new inode. This can be seen in the following example where a file is copied from a flash drive mounted on a Windows client’s file system under drive ‘E:’, to a cluster’s SMB share mounted at drive ‘Z:’.

The ‘Date created’ date above is ahead in time of both the ‘accessed’ and ‘modified’, because the latter two were merely inherited from the source file, whereas the create date was set when the copy was made.

The corresponding ‘date’, ‘stat’, and ‘isi get’ CLI output from the cluster confirms this:

In before OneFS 9.5, when a file is replicated, its create date is timestamped when that file was copied from the source cluster. This means when the replication job ran, or, more specifically, when the individual job worker thread got around to processing that specific file.

By way of contrast, OneFS 9.5 now ensures that SyncIQ fully replicates the full array of metadata, preserving all values, including that of the birth time / create date.

The primary consideration for the new create date functionality is that it requires both source and target clusters in a replication set to be running OneFS 9.5 or later.

If either the source or the target is running pre-9.5 code, this time field retains its old behavior of being set to the time of replication (actual file creation) rather than the correct value associated with the source file.

In OneFS 9.5, create date timestamping works exactly the same way as SyncIQ replication of other metadata (such as ‘mtime’, etc), occurring automatically as part of every file replication. Plus, no additional configuration is necessary beyond upgrading both clusters to OneFS 9.5.

One other significant thing to note about this feature is that SyncIQ is changelist-based, using OneFS snapshots under the hood for its checkpointing and delta comparisons.. This means that, if a replication relationship has been configured prior to OneFS 9.5 upgrade, the source cluster will have valid birthtime data, but the target’s birthtime data will reflect the local creation time of the files it’s copied.

Note that, upon upgrading both sides to OneFS 9.5 and running a SyncIQ job, nothing will change. This is  because SyncIQ will perform its snapshot comparison, determine that no changes were made to the dataset, and so will not perform any replication work. However, if a source file is ‘touched’ so that it’s mtime is changed (or any other action performed that will cause a copy-on-write, or CoW) that will cause the file to show up in the snapshot diff and therefore be replicated. As part of replicating that file, the correct birth time will be written on the target.

Note that a full replication (re)sync does not get triggered as a result of upgrading a replication cluster pair to OneFS 9.5 or later and thereby enabling this functionality. Instead, any create date timestamp resolution happens opportunistically and in the background as files gets touched or modified – and thereby replicated. Be aware that ‘touching’ a file does change its modification time, in addition to updating the create date, which may be undesirable.

In the previous article in this series we looked at the architecture and operation of OneFS configuration auditing. Now, we’ll turn our attention to its management, event viewing, and troubleshooting.

The CLI command set for configuring ‘isi audit’ is split between two functional areas:

Area	Detail	Syntax
Events	Specifies which specific events get logged, across three categories: ·         Audit Failure ·         Audit Success ·         Syslog Audit Events	isi audit settings …
Global	Configuration of global audit parameters, including topics, zones, CEE, syslog, puring, retention, etc.	isi audit settings global …

The ‘view’ argument for each command returns the following output:

1. Events:

# isi audit settings view

            Audit Failure: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

            Audit Success: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

      Syslog Audit Events: create_file, create_directory, open_file_write, open_file_read, close_file_unmodified, close_file_modified, delete_file, delete_directory, rename_file, rename_directory, set_security_file, set_security_directory

Syslog Forwarding Enabled: No

2. Global:

# isi audit settings global view

     Protocol Auditing Enabled: Yes

                 Audited Zones: -

               CEE Server URIs: -

                      Hostname:

       Config Auditing Enabled: Yes

         Config Syslog Enabled: No

         Config Syslog Servers: -

     Config Syslog TLS Enabled: No

  Config Syslog Certificate ID:

       Protocol Syslog Servers: -

   Protocol Syslog TLS Enabled: No

Protocol Syslog Certificate ID:

         System Syslog Enabled: No

         System Syslog Servers: -

     System Syslog TLS Enabled: No

  System Syslog Certificate ID:

          Auto Purging Enabled: No

              Retention Period: 180

       System Auditing Enabled: No

While configuration auditing is disabled on OneFS by default, the following CLI syntax can be used enable and verify config auditing across the cluster:

# isi audit settings global modify --config-auditing-enabled 1

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Similarly, to enable configuration change audit redirection to syslog:

# isi audit settings global modify --config-auditing-enabled true

# isi audit settings global modify --config-syslog-enabled true

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

Or to disable redirection to syslog:

# isi audit settings global modify --config-syslog-enabled false

# isi audit settings global modify --config-auditing-enabled false

Up to six CEE servers per cluster can be configured. For example:

#isi audit settings global modify --add-cee-server-uris='<URL>’ For example: #isi audit settings global modify --add-cee-server-uris='http://cee1.isilon.com:12228/cee'

Auditing can be constrained by access zone, too:

# isi audit settings modify --add-audited-zones=audit_az1

Note that, when auditing is enabled, the system zone is included by default. However, it can be excluded, if desired:

# isi audit setting modify --remove-audited-zones=System

Access zone’s audit parameters can also be configured via the ‘isi zones’ CLI command set. For example:

# isi zone zones create --all-auth-providers=true --audit-failure=all --audit-success=all --path=/ifs/data --name=audit_az1

Granular audit event type configuration can be specified, if desired, to narrow the scope and reduce the amount of audit logging, etc.

For example, the following command syntax constrains auditing to read and logon failures and successful writes and deletes under path /ifs/data in the audit_az1 access zone:

# isi zone zones create --all-auth-providers=true --audit-failure=read,logon --audit-success=write,delete --path=/ifs/data --name=audit_az1

In addition to the CLI, the OneFS platform API can also be used to configure and manage auditing. For example, to enable configuration auditing on a cluster:

PUT /platform/1/audit/settings

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

        {

               'config_auditing_enabled': True

        }

The following ‘204’ HTTP response code from the cluster indicates that the request was successful, and that configuration auditing is now enabled on the cluster. No message body is returned for this request.

204 No Content

Content-type: text/plain,

Allow: 'GET, PUT, HEAD'

Similarly, to modify the config audit topic’s maximum cached messages threshold to a value of ‘1000’ via the API:

PUT /1/audit/topics/config

Authorization: Basic QWxhZGRpbjpvcGVuIHN1c2FtZQ==

    {

        "max_cached_messages": 1000

    }

Again, no message body is returned from OneFS for this request.

204 No Content

Content-type: text/plain,

Allow: 'GET, PUT, HEAD'

Note that, in the unlikely event that a cluster experiences an outage during which it loses quorum, auditing will be suspended until it is regained. Events similar to the following will be written to the /var/log/audit_d.log file:

940b5c700]: Lost quorum! Audit logging will be disabled until /ifs is writeable again.

2023-08-28T15:37:32.132780+00:00 <1.6> TME-1(id1) isi_audit_d[6495]: [0x345940b5c700]: Regained quorum. Logging resuming.

When it comes to reading audit events on the cluster, OneFS natively provides the handy ‘isi_audit_viewer’ utility. For example, the following audit viewer output shows the events logged when the cluster admin added the ‘/ifs/tmp’ path to the SmartDedupe configuration, and created a new user named ‘test’1’:

# isi_audit_viewer

[0: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076315499,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/1/dedupe/settings","method":"PUT","args":{}

,"body":{"paths":["/ifs/tmp"]}

}}

[1: Tue Aug 29 23:01:16 2023] {"id":"f54a6bec-46bf-11ee-920d-0060486e0a26","timestamp":1693350076391422,"payload":{"status":204,"statusmsg":"No Content","body":{}}}

[2: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223446993,"payload":{"user":{"token": {"UID":0, "GID":0, "SID": "SID:S-1-22-1-0", "GSID": "SID:S-1-22-2-0", "GROUPS": ["SID:S-1-5-11", "GID:5", "GID:10", "GID:20", "GID:70"], "protocol": 17, "zone id": 1, "client": "10.135.6.255", "local": "10.219.64.11" }},"uri":"/18/auth/users","method":"POST","args":{}

,"body":{"name":"test1"}

}}

[3: Tue Aug 29 23:03:43 2023] {"id":"4cfce7a5-46c0-11ee-920d-0060486e0a26","timestamp":1693350223507797,"payload":{"status":201,"statusmsg":"Created","body":{"id":"SID:S-1-5-21-593535466-4266055735-3901207217-1000"}

}}

The audit log entries, such as those above, typically comprise the following components:

Order	Component	Detail
1	Timestamp	Timestamp in human readable form
2	ID	Unique entry ID
3	Timestamp	Timestamp in UNIX epoch time
4	Node	Node number
5	User tokens	1.       The user tokens of the Roles and rights of user executing the command ·         User persona (Unix/Windows) ·         Primary group persona (Unix/Windows) ·         Supplemental group personas (Unix/Windows) ·         RBAC privileges of the user executing the command
6	Interface	Interface used to generate the command: ·         10 = pAPI / WebUI ·         16 = Console CLI ·         17 = SSH CLI
7	Zone	Access zone that the command was executed against
8	Client IP	Where the user connected from
9	Local node	Local node address where the command was executed
10	Command	Command syntax
11	Arguments	Command arguments
12	Body	Command body

The ‘isi_audit_viewer’ utility automatically reads the ‘config’ log topic by default, but can also be used read the ‘protocol’ log topic too. Its CLI command syntax is as follows:

# isi_audit_viewer -h

Usage: isi_audit_viewer [ -n <nodeid> | -t <topic> | -s <starttime>|

         -e <endtime> | -v ]

         -n <nodeid> : Specify node id to browse (default: local node)

         -t <topic>  : Choose topic to browse.

            Topics are "config" and "protocol" (default: "config")

         -s <start>  : Browse audit logs starting at <starttime>

         -e <end>    : Browse audit logs ending at <endtime>

         -v verbose  : Prints out start / end time range before printing records

Note that, on large clusters where there is heavy (ie. 100,000’s) of audit writes, when running the isi_audit_viewer utility across the cluster with ‘isi_for_array’, it can potentially lead to memory starvation and other issues – especially if outputting to a directory under /ifs. As such, consider directing the output to a non-IFS location such as /var/temp. Also, the isi_audit_viewer ‘-s’ (start time) and ‘-e’ (end time) flags can be used to limit a search (ie. for  1-5 minutes), helping reduce the size of data.

In addition to reading audit events, the view is also a useful tool to assist with troubleshoot any auditing issues. Additionally, any errors that are encountered while processing audit events, and when delivering them to an external CEE server, are written to the log file ‘/var/log/isi_audit_cee.log’. Additionally, the protocol specific logs will contain any issues the audit filter has collecting while auditing events.

OneFS auditing can detect potential sources of data loss, fraud, inappropriate entitlements, access attempts that should not occur, and a range of other anomalies that are indicators of risk. This can be especially useful when the audit associates data access with specific user identities.

In the interests of data security, OneFS provides ‘chain of custody’ auditing by logging specific activity on the cluster. This includes OneFS configuration changes plus NFS, SMB, and HDFS client protocol activity, which are required for organizational IT security compliance, as mandated by regulatory bodies like HIPAA, SOX, FISMA, MPAA, etc.

OneFS auditing uses Dell’s Common Event Enabler (CEE) to provide compatibility with external audit applications. A cluster can write audit events across up to five CEE servers per node in a parallel, load-balanced configuration. This allows OneFS to deliver an end to end, enterprise grade audit solution which efficiently integrates with third party solutions like Varonis DatAdvantage.

The following diagram outlines the basic architecture of OneFS audit:

Both system configuration changes, as well as protocol activity, can be easily audited on a PowerScale cluster. However, the protocol path is greyed out above, since it is outside the focus of this article. More information on OneFS protocol auditing can be found here.

As illustrated above, the OneFS audit framework is centered around three main services.

Service	Description
isi_audit_cee	Service allowing OneFS to support third-party auditing applications. The main method of accessing protocol audit data from OneFS is through a third-party auditing application.
isi_audit_d	Responsible for per-node audit queues and managing the data store for those queues. It provides a protocol on which clients may produce event payloads within a given context. It establishes a Unix domain socket for queue producers and handles writing and rotation of log files in /ifs/.ifsvar/audit/logs/node###/{config,protocol}/*.
isi_audit_syslog	Daemon providing forwarding of audit config and protocol events to syslog.

The basic configuration auditing workflow sees a cluster config change request come in via either the OneFS CLI, WebUI or platform API. The API handler infrastructure passes this request to the isi_audit_d service, which intercepts it as a client thread and adds it to the audit queue. It is then processed and passed via a backend thread and written to the audit log files (IFS) as appropriate.

If audit syslog forwarding has been configured, IFS also passes the event to the isi_audit_syslog daemon, where a supervisor process instructs a writer thread to send it to the syslog, which in turn updates its pertinent /var/log/ logfiles.

Similarly, if Common Event Enabler (CEE) forwarding has been enabled, IFS will also pass the request to the isi_audit_cee service, where a delivery worker threads will intercept it and send the event to the CEE server pool. The isi_audit_cee heartbeat task makes CEE servers available for audit event delivery. Only after a CEE server has received a successful heartbeat will audit events be delivered to it. Every ten seconds the heartbeat task wakes up and sends each CEE server in the configuration a heartbeat. While CEE servers are available and events are in memory an attempt will be made to deliver these. Shutdown will only save audit log position if all the events are delivered to CEE since audit should not lose events. It isn’t critical that all events are delivered at shutdown since any unsaved events can be resent to CEE on the next start of isi_audit_cee since CEE handles duplicates.

Within OneFS, all audit data is organized by topic and is securely stored in the file system.

# isi audit topics list

Name     Max Cached Messages

-----------------------------

protocol 2048

config   1024

-----------------------------

Total: 2

Auditing can detect a variety of potential sources of data loss. These include unauthorized access attempts, inappropriate entitlements, plus a bevy of other fraudulent activities that plague organizations across the gamut of industries. Enterprises are increasingly tasked required to comply with stringent regulatory mandates developed to protect against these sources of data theft and loss.

OneFS system configuration auditing is designed to track and record all configuration events that are handled by the API through the command-line interface (CLI).

# isi audit topics view config

               Name: config

Max Cached Messages: 1024

Once enabled, system configuration auditing requires no additional configuration, and auditing events are automatically stored in the config audit topic directories. Audit access and management is governed by the ‘ISI_PRIV_AUDIT’ RBAC privilege, and OneFS provides a default ‘AuditAdmin’ role for this purpose.

Audit events are stored in a binary file under /ifs/.ifsvar/audit/logs. The logs automatically roll over to a new file after the size reaches 1 GB. The audit logs are consumable by auditing applications that support the Dell Common Event Enabler (CEE).

OneFS audit topics and settings can easily be viewed and modified. For example, to increase the configuration auditing maximum cached messages threshold to 2048 from the CLI:

# isi audit topics modify config --max-cached-messages 2048

# isi audit topics view config

               Name: config

Max Cached Messages: 2048

Audit configuration can also be modified or viewed per access zone and/or topic.

Operation	CLI Syntax	Method and URI
Get audit settings	isi audit settings  view	GET <cluster-ip:port>/platform/3/audit/settings
Modify audit settings	isi audit settings modify …	PUT <cluster-ip:port>/platform/3/audit/settings
View JSON schema for this resource, including query parameters and object properties info.		GET <cluster-ip:port>/platform/3/audit/settings?describe
View JSON schema for this resource, including query parameters and object properties info.		GET <cluster-ip:port>/platform/1/audit/topics?describe

Configuration auditing can be enabled on a cluster from either the CLI or platform API. The current global audit configuration can be viewed as follows:

# isi audit settings global view

     Protocol Auditing Enabled: No

                 Audited Zones: -

               CEE Server URIs: -

                      Hostname:

       Config Auditing Enabled: No

         Config Syslog Enabled: No

         Config Syslog Servers: -

     Config Syslog TLS Enabled: No

  Config Syslog Certificate ID:

       Protocol Syslog Servers: -

   Protocol Syslog TLS Enabled: No

Protocol Syslog Certificate ID:

         System Syslog Enabled: No

         System Syslog Servers: -

     System Syslog TLS Enabled: No

  System Syslog Certificate ID:

          Auto Purging Enabled: No

              Retention Period: 180

       System Auditing Enabled: No

In this case, configuration auditing is disabled – its default setting. The following CLI syntax will enable (and verify) configuration auditing across the cluster:

# isi audit settings global modify --config-auditing-enabled 1

# isi audit settings global view | grep -i 'config audit'

       Config Auditing Enabled: Yes

In the next article, we’ll look at the config audit management, event viewing, and troubleshooting.