SyncIQ receives an update in OneFS 9.11 with the default enablement of its Temporary Directory Hashing feature, which can help improve replication directory delete performance on target clusters.
But first, some background. For several years now, OneFS has included functionality, commonly referred to as temporary directory hashing, which addresses some of the challenges that SyncIQ can potentially encounter with large incremental replication tasks. Specifically, if a cluster contains an extra-wide directory, with many different replication threads trying to write to it simultaneously, OneFS file system performance can be impacted due to contention over lock requests on that very wide directory.
When Sync IQ performs an incremental transfer it frequently uses a temporary working directory in cases such as where a file has been created, but its parent doesn’t exist yet due to LIN order processing – or files being removed and their parents not being available, etc. Instead, SyncIQ will use this temporary working directory as a place to stash these files until it can put them in the correct location. In some incremental replication workflows, this could result in an extra-wide temporary working directory, potentially containing millions or billions of directories entries. When there are hundreds of SyncIQ workers all trying to link and unlink files from that same directory, performance can start to become impacted.
To address this, temporary directory hashing introduces support for subdirectories within a large temp working directory, based on a directory cookie. This allows SyncIQ to split up that monolithic directory into a bunch of smaller ones, so workers don’t contend with all of the other workers when they’re trying to link and unlink files within their temporary directories. They only contend with the other workers in their particular subdirectory, often providing a significant performance boost in some cases with lots of concurrent access.
In OneFS 9.11, temporary directory hashing functionality now becomes the default configuration and behavior.
SyncIQ’s temporary directory hashing functionality has actually existed within OneFS since 8.2, but prior to OneFS 9.11 it had to be manually enabled on a per-policy basis for any desired replication workflows.
When installing or upgrading a cluster to OneFS 9.11 or later, temporary directory hashing becomes the default configuration, so that any new SyncIQ policies will automatically have temporary directory hashing will enabled. However, this default change will not be applied retroactively to any legacy policies that were configured prior to OneFS 9.11 upgrade.
That said, any pre-existing policies can be easily configured to use temporary directory hashing from the SyncIQ source cluster with the following CLI syntax:
# echo '{"enable_hash_tmpdir": true}' | isi_papi_tool PUT /7/sync/policies/<policy name>
For example:
# echo '{"enable_hash_tmpdir": true}' | isi_papi_tool PUT /7/sync/policies/remote_zone1
Type request body, press enter, and CTRL-D:
204
Content-type: text/plain
Allow: GET, PUT, DELETE, HEAD
The configuration can be verified as ‘enabled’ with the following command:
# isi_papi_tool GET /7/sync/policies/remote_zone1 | grep "enable_hash_tmpdir"
"enable_hash_tmpdir" : true,
Under the hood, temporary directory hashing places any directories within a SyncIQ policy which need to be deleted into subdirectories under the ./tmp-working-dir/ directory, instead at the root of tmp-working-dir. This lowers contention on the root tmp-working-dir by moving exclusive locking requests to those subdirectories.
Performance-wise, the benefit and efficiency of SyncIQ temporary directory hashing will vary by cluster constitution, environment, and workflow. However, environments with thousands of directory deletions per policy run have seen improvements of between 2x-20x faster delete performance. To determine whether this feature is proving beneficial for a specific policy, view the SyncIQ job reports and compare the ‘STF_PHASE_CT_DIR_DELS’ job phase start and end times. This will indicate how much time those jobs have spent in this temporary directory delete phase, and can be accomplished from the replication source cluster with the following CLI syntax:
# isi sync reports view <policy_name> <job_id> | grep -C 3 "CT_DIR_DELS"
For example:
# isi sync reports view remote_zone1 31 | grep -C 3 "CT_DIR_DELS" Phase: STF_PHASE_CT_DIR_DELS Start Time: 2025-06-06T16:12:39 End Time: 2025-06-06T16:10:47
Note that for some SyncIQ policies which routinely move wide and shallow directories from one directory to another, temporary directory hashing may actually adversely impact those moves. In such instances, the feature can be disabled for each individual SyncIQ replication policy as follows:
# echo '{"enable_hash_tmpdir": false}' | isi_papi_tool PUT /7/sync/policies/<policy name>
Note that the above command should be run on the replication source cluster, using the root user authenticating to the PAPI service, replacing <policy name> with the appropriate value.
For example:
# echo '{"enable_hash_tmpdir": false}' | isi_papi_tool PUT /7/sync/policies/remote_zone1
Type request body, press enter, and CTRL-D:
204
Content-type: text/plain
Allow: GET, PUT, DELETE, HEAD
As such, OneFS will have disabled this feature for all subsequent job runs.
Similarly, the configuration can be verified as follows:
# isi_papi_tool GET /7/sync/policies/remote_zone1 | grep "enable_hash_tmpdir"
"enable_hash_tmpdir" : false,
For clusters running OneFS 8.2 through OneFS 9.10, where SyncIQ temporary directory deletion is disabled by default, it can be activated on a per-policy basis as follows:
# echo '{"enable_hash_tmpdir": true}' | isi_papi_tool PUT /7/sync/policies/<policy name>
As such, the next time SyncIQ runs the specified policy, temporary directory hashing will be enabled for this and future job runs.
So, in summary, SyncIQ temporary directory hashing can improve SyncIQ directory deletion performance for many policies with wide directory cases. While in OneFS 9.10 and earlier it was manually configurable on an individual per policy basis, now, in OneFS 9.11 and later, temporary directory hashing is enabled by default on all new Sync IQ policies.