OneFS Instant Delete

Another interesting component of the recent OneFS 9.4 release is the new ‘instant delete’ feature. A variety of workflows often result in the create large, deep directory structures containing a large amount of data. Deleting these extensive trees can be very slow, particularly if they contain many files, and typically the client needs to remain connected to the cluster for the duration of the deletion process. However, with OneFS instant delete, the global trash directory, or GTD, can remove the data asynchronously, effectively eliminating the delete operation from the processing path.

This instant delete functionality in its first version provides both SmartSync and Writable Snapshots with the ability to rapidly delete a directory from their replication and snapshot paths respectively.

Under the hood, the OneFS instant delete v1 workflow is architected as follows:

OneFS instant delete is quarterbacked by the isi_trash_d daemon, and leverages the OneFS job engine service and its TreeDelete job, as well as the persistent queue service (PQ) for SmartSync and writable snapshot support.  The global trash directory (/ifs/.isi_trash_bin) is created by default after a cluster is upgraded to OneFS version 9.4. A new OneFS static attribute, ‘di_istrashdir’, indicates whether a directory is a GTD. Once created, a GTD cannot be deleted.

The isi_trash_d daemon, under the control of MCP, continuously runs in the background and monitors the GTD for new deletion requests. By default, this /ifs/.isi_trash_bin directory or its contents are not visible to end users. Unwanted directories can be deleted by either moving or renaming them to the GTD, and the ‘ISI_PRIV_JOB_ENGINE’ RBAC privilege is required for this. Once a garbage directory is moved to the trash directory, it is not visible in the next snapshot.

So let’s take a look at the process at work. In the example below, the /ifs/temp directory contains a number of files and directories that are no longer required. To delete this data, all that’s required is to move this directory to the global trash directory, resulting in the entire subtree being asynchronously deleted.

# mv /ifs/temp /ifs/.ifs_trash_bin

The isi_trash_d daemon runs in the background, continually waiting for delete-able contents to appear in the trash bin:

# ps -auxw | grep trash

root   11569    0.0  0.0  22828   7108  -  Is   Mon18       0:00.00 /usr/sbin/isi_trash_d

Once the directory move command has been executed, the isi_trash_d daemon intercepts the request and renames the directory, in this case to ‘/ifs/.isi_trash_bin/temp135b41f51’:

The isi_trash_d daemon then automatically kicks off a parallel TreeDelete job to remove the /ifs/test directory:

# isi job jobs list

ID   Type               State   Impact  Policy  Pri  Phase  Running Time


139  TreeDelete         Running Medium  MEDIUM  4    1/1    -


Total: 1

The TreeDelete job’s details show the removal of the ‘/ifs/.isi_trash_bin/temp135b41f51’ directory:

# isi job jobs view 139

               ID: 139

             Type: TreeDelete

            State: Running

           Impact: Medium

           Policy: MEDIUM

              Pri: 4

            Phase: 1/1

       Start Time: 2022-08-03T20:45:22

     Running Time: 44s

     Participants: 1, 2, 3, 4, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 45, 46, 48, 51, 53, 55, 56, 57, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109

         Progress: Processed 186 files and 3 directories, approx. 9498 MB, 14088528 blocks; 0 errors

Waiting on job ID: -

      Description: {'count': 1, 'lins': {'1:35b4:1f51': """/ifs/.isi_trash_bin/temp135b41f51"""}}

       Human Desc:

# isi job report view 139

TreeDelete[139] phase 1 (2022-08-03T20:48:43)


Paths           [ "/ifs/.isi_trash_bin/temp135b41f51" ]

Files           4922

Directories     100

Apparent size   263054252151

Physical size   192112713728

JE/Error Count  0

JE/Time elapsed 199 seconds (3m19s)

JE/Time working 199 seconds (3m19s)

TreeDelete[139] Job Summary


Final Job State  Succeeded

Phase Executed   1

By default, if a TreeDelete job does not complete within 72 hours, the job is canceled by isi_trash_d daemon monitoring the job progress.

# isi_gconfig -t trash-config

[root] {version:1}

min_job_queue_interval (uint64) = 30

job_query_interval (uint64) = 30

job_timeout_limit (uint64) = 259200

hc_alert_pq_limit (uint64) = 10240

From above output, we can see that the job_timeout_limit is set by default to 72 hours  (259200 seconds) and the job_query_interval to 30 seconds.

When it comes to troubleshooting, the isi_trash_d daemon writes to three log files, under the purview of OneFS ilog:

  • /var/log/isi_trash_d.log
  • /var/log/messages
  • /var/log/isi_trash_pq_clean.log

By default, logging verbosity is at the ‘info+’ level, but can be easily adjusted using the isi_ilog CLI utility:

# isi_ilog -a isi_trash_d --level debug

Logging change for isi_trash_d

          use_syslog: no change

          use_stderr: no change

            log_file: no change

               level: info+  -> debug

    syslog_threshold: no change

       log_thread_id: no change

                tags: no change

While instant delete is enabled by default in OneFS 9.4, it can also be disabled if desired by disabling the isi_trash_d service:

# isi services -a isi_trash_d disable

Or by setting the trash_delete_enabled sysctl to value ‘0’:

# sysctl efs.gtd.trash_delete_enabled=0

Additionally, the hiding visibility of the GTD can also be controlled by the ‘hide_trash_dir’ sysctl:

# sysctl efs.gtd.hide_trash_dir

As mentioned previously, since OneFS 9.4 introduces first version of instant delete, there are some caveats and limitations to be aware of. These include:

  • A move to the GTD is not permitted if a QuotaScan job is running and the garbage directory is part of the quota domain.
  • SmartLock domain restrictions do not allow moving a garbage directory out of a WORM domain.
  • Only directories, and not individual files, may be placed in the GTD:
# ls -lsia f1

5204682280 24 -rw-------     1 root  wheel  0 Aug  3 21:49 f1

# mv f1 /ifs/.isi_trash_bin

mv: rename f1 to /ifs/.isi_trash_bin/f1: Not a directory
  • The ‘isi get’ command is also unable to see the GTD or its contents:
# isi get -DD /ifs/.ifs_trash_bin

isi: /ifs/.ifs_trash_bin: No such file or directory
  • Despite being ‘hidden’ by default, the GTD is able to be exported via OneFS protocols.
  • HDFS in some instances does not allow a rename.


Leave a Reply

Your email address will not be published. Required fields are marked *