Another interesting component of the recent OneFS 9.4 release is the new ‘instant delete’ feature. A variety of workflows often result in the create large, deep directory structures containing a large amount of data. Deleting these extensive trees can be very slow, particularly if they contain many files, and typically the client needs to remain connected to the cluster for the duration of the deletion process. However, with OneFS instant delete, the global trash directory, or GTD, can remove the data asynchronously, effectively eliminating the delete operation from the processing path.
This instant delete functionality in its first version provides both SmartSync and Writable Snapshots with the ability to rapidly delete a directory from their replication and snapshot paths respectively.
Under the hood, the OneFS instant delete v1 workflow is architected as follows:
OneFS instant delete is quarterbacked by the isi_trash_d daemon, and leverages the OneFS job engine service and its TreeDelete job, as well as the persistent queue service (PQ) for SmartSync and writable snapshot support. The global trash directory (/ifs/.isi_trash_bin) is created by default after a cluster is upgraded to OneFS version 9.4. A new OneFS static attribute, ‘di_istrashdir’, indicates whether a directory is a GTD. Once created, a GTD cannot be deleted.
The isi_trash_d daemon, under the control of MCP, continuously runs in the background and monitors the GTD for new deletion requests. By default, this /ifs/.isi_trash_bin directory or its contents are not visible to end users. Unwanted directories can be deleted by either moving or renaming them to the GTD, and the ‘ISI_PRIV_JOB_ENGINE’ RBAC privilege is required for this. Once a garbage directory is moved to the trash directory, it is not visible in the next snapshot.
So let’s take a look at the process at work. In the example below, the /ifs/temp directory contains a number of files and directories that are no longer required. To delete this data, all that’s required is to move this directory to the global trash directory, resulting in the entire subtree being asynchronously deleted.
# mv /ifs/temp /ifs/.ifs_trash_bin
The isi_trash_d daemon runs in the background, continually waiting for delete-able contents to appear in the trash bin:
# ps -auxw | grep trash root 11569 0.0 0.0 22828 7108 - Is Mon18 0:00.00 /usr/sbin/isi_trash_d
Once the directory move command has been executed, the isi_trash_d daemon intercepts the request and renames the directory, in this case to ‘/ifs/.isi_trash_bin/temp135b41f51’:
The isi_trash_d daemon then automatically kicks off a parallel TreeDelete job to remove the /ifs/test directory:
# isi job jobs list ID Type State Impact Policy Pri Phase Running Time ------------------------------------------------------------------------- 139 TreeDelete Running Medium MEDIUM 4 1/1 - ------------------------------------------------------------------------- Total: 1
The TreeDelete job’s details show the removal of the ‘/ifs/.isi_trash_bin/temp135b41f51’ directory:
# isi job jobs view 139
ID: 139
Type: TreeDelete
State: Running
Impact: Medium
Policy: MEDIUM
Pri: 4
Phase: 1/1
Start Time: 2022-08-03T20:45:22
Running Time: 44s
Participants: 1, 2, 3, 4, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 45, 46, 48, 51, 53, 55, 56, 57, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109
Progress: Processed 186 files and 3 directories, approx. 9498 MB, 14088528 blocks; 0 errors
Waiting on job ID: -
Description: {'count': 1, 'lins': {'1:35b4:1f51': """/ifs/.isi_trash_bin/temp135b41f51"""}}
Human Desc:
# isi job report view 139
TreeDelete[139] phase 1 (2022-08-03T20:48:43)
---------------------------------------------
Paths [ "/ifs/.isi_trash_bin/temp135b41f51" ]
Files 4922
Directories 100
Apparent size 263054252151
Physical size 192112713728
JE/Error Count 0
JE/Time elapsed 199 seconds (3m19s)
JE/Time working 199 seconds (3m19s)
TreeDelete[139] Job Summary
---------------------------
Final Job State Succeeded
Phase Executed 1
By default, if a TreeDelete job does not complete within 72 hours, the job is canceled by isi_trash_d daemon monitoring the job progress.
# isi_gconfig -t trash-config
[root] {version:1}
min_job_queue_interval (uint64) = 30
job_query_interval (uint64) = 30
job_timeout_limit (uint64) = 259200
hc_alert_pq_limit (uint64) = 10240
From above output, we can see that the job_timeout_limit is set by default to 72 hours (259200 seconds) and the job_query_interval to 30 seconds.
When it comes to troubleshooting, the isi_trash_d daemon writes to three log files, under the purview of OneFS ilog:
- /var/log/isi_trash_d.log
- /var/log/messages
- /var/log/isi_trash_pq_clean.log
By default, logging verbosity is at the ‘info+’ level, but can be easily adjusted using the isi_ilog CLI utility:
# isi_ilog -a isi_trash_d --level debug Logging change for isi_trash_d use_syslog: no change use_stderr: no change log_file: no change level: info+ -> debug syslog_threshold: no change log_thread_id: no change tags: no change
While instant delete is enabled by default in OneFS 9.4, it can also be disabled if desired by disabling the isi_trash_d service:
# isi services -a isi_trash_d disable
Or by setting the trash_delete_enabled sysctl to value ‘0’:
# sysctl efs.gtd.trash_delete_enabled=0
Additionally, the hiding visibility of the GTD can also be controlled by the ‘hide_trash_dir’ sysctl:
# sysctl efs.gtd.hide_trash_dir
As mentioned previously, since OneFS 9.4 introduces first version of instant delete, there are some caveats and limitations to be aware of. These include:
- A move to the GTD is not permitted if a QuotaScan job is running and the garbage directory is part of the quota domain.
- SmartLock domain restrictions do not allow moving a garbage directory out of a WORM domain.
- Only directories, and not individual files, may be placed in the GTD:
# ls -lsia f1 5204682280 24 -rw------- 1 root wheel 0 Aug 3 21:49 f1 # mv f1 /ifs/.isi_trash_bin mv: rename f1 to /ifs/.isi_trash_bin/f1: Not a directory
- The ‘isi get’ command is also unable to see the GTD or its contents:
# isi get -DD /ifs/.ifs_trash_bin isi: /ifs/.ifs_trash_bin: No such file or directory
- Despite being ‘hidden’ by default, the GTD is able to be exported via OneFS protocols.
- HDFS in some instances does not allow a rename.