OneFS audit logs can grow quickly based on a customer’s audit configuration. As such, audit logs often need to be trimmed for space or for regulatory compliance purposes. OneFS 9.1 introduces the ability to automatically and non-disruptively purge both configuration and protocol audit log files. In releases prior to OneFS 9.1, there was no easy way to remove audit logs without stopping and restarting the audit service. Additionally, it involved a somewhat complex and potentially error-prone procedure. The steps must be followed explicitly, otherwise it may lead to data unavailability. OneFS audit logs are stored and compressed binary files which are located under /ifs/.ifsvar/audit/logs, and each log file can grow to a maximum of 1 GB.
Audit log purging provides a simple, efficient method to trim audit log entries. Log purging is applied to all nodes in the cluster, either automatically and periodically based on a retention policy, or manually. Configuration is via the OneFS CLI or platform API, and an automated purging policy is based on a time range, or ‘retentionperiod(by days)’. When run automatically, the log purger runs once an hour as a background process with minimal performance impact.
The OneFS 9.1 ‘isi audit settings’ command set enables the configuration and control of automatic log purging. The configuration parameters can be viewed with the following syntax:
# isi audit settings global view Protocol Auditing Enabled: No Audited Zones: - CEE Server URIs: - Hostname: Config Auditing Enabled: No Config Syslog Enabled: No Config Syslog Servers: - Protocol Syslog Servers: - Auto Purging Enabled: No Retention Period: 180
The following CLI command will enable automatic purging:
# isi audit settings global modify --auto-purging-enabled=yes You are enabling the automatic log purging. Automatic log purging will run in background to delete audit log files. Please check the retention period before enabling automatic log purging. Are you sure you want to do this?? (yes/[no]): yes
This change can be verified with the following syntax:
# isi audit settings global view | grep -i purging Auto Purging Enabled: Yes
Similarly, the retention period can also be configured from the OneFS CLI as follows. In this case, it’s being changed to 90 days from its default of 180 days:
# isi audit settings global modify --retention-period=90 # isi audit settings global view | grep -i retention Retention Period: 365
Manual deletion uses the same underlying mechanism as auto deletion, the only difference being that it is initiated manually. The isi audit logs delete CLI command can be used to purge the audit logs prior to a specified date. For example, the following syntax will manually purge the audit logs of entries prior to 1st April 2020:
# isi audit logs delete --before=2020-04-01 You are going to delete the audit logs before 2020-04-01. Are you sure you want to do this?? (yes/[no]): yes The purging request has been triggered.
The following CLI command can be run to monitor and verify the activity of the manual purging process:
# isi audit logs check Purging Status: Using Before Value: 2019-10-01 Currently Manual Purging Status: COMPLETED
The removal of protocol or configuration audit logs for a particular time period can be verified with the OneFS audit event viewer command line utility. For example, the following syntax will display any protocol audit events between 1st January 2020 and 31st March 2019:
# isi_audit_viewer -t protocol -s "2020-01-01 12:00:00" -e "2020-03-31 12:00:00"
There are a couple of things to keep in mind when configuring and using audit logfile purging:
If syslog forwarding is enabled and is unable to forward quick enough, it can potentially block the purging process. This could potentially result in remaining, un-purged audit log entries, even if they fall outside of the retention period. The same bottlenecking challenge also could potentially occur with the CEE forwarder.
Performance-wise, depending on the quantity of audit data to be purged, the initial deletion may be lengthy and somewhat resource intensive. However, subsequent deletions will typically be much faster and with negligible performance impact.
The following CLI commands can be useful in determining how logging in general is progressing on a cluster. The ‘isi_audit_progress’ syntax will display both last consumed and logged event timestamps. Note that the command is node-local, so will need to be run with ‘isi_for_array’ to get a full cluster report:
# isi_for_array -s isi_audit_progress -t protocol CEE_FWD tme-1: Last consumed event time: '2022-12-02 10:16:24' tme-1: Last logged event time: '2022-12-02 10:16:27' tme-2: Last consumed event time: '2022-12-02 10:16:19' tme-2: Last logged event time: '2022-12-02 10:16:24' tme-3: Last consumed event time: '2022-12-02 10:16:31' tme-3: Last logged event time: '2022-12-02 10:16:31'
Next, the following CLI command will confirm that nodes are happily forwarding to the CEE server(s):
# isi statistics query current list --keys=node.audit.cee.export.rate --nodes all Node node.audit.cee.export.rate ----------------------------------- 1 107.200000 2 89.400000 3 94.100000 average 96.900000 -----------------------------------