Data replication is only half the equation. The other part, which often gets tested at the worst possible moment, is what happens when the primary cluster goes dark and storage administrators need to redirect clients to the secondary site with minimum disruption and maximum confidence in data integrity.
OneFS 9.15 introduces SmartSync failover and failback capabilities to ensure uninterrupted operations and fast recovery when a primary PowerScale cluster becomes unavailable. Rather than relying on time‑consuming restores from traditional backups, SmartSync enables rapid redirection of client access to a secondary cluster. By keeping data and metadata immediately available, it significantly reduces Recovery Time Objectives (RTO).
When the primary cluster is restored, SmartSync efficiently performs failback by syncing only the incremental changes generated during the failover period, preserving data integrity and enabling a seamless return to normal replication operations.
Together, these capabilities deliver a robust, high‑availability data mobility solution that strengthens business continuity and lowers operational risk.
SmartSync Failover/Failback provides the following capabilities:
· Converts a read‑only SmartSync target path into a writeable path.
· Switches a writeable SmartSync source path to read‑only.
· Restores a SmartSync base path to a previous dataset state.
· Enables reverse incremental synchronization to transfer client‑written data from a failover cluster back to the original source.
With OneFS SyncIQ, the allow-writes > resync-prep > redirect > reverse flow is generally well understood. In contrast, OneFS SmartSync has some notable variations to this theme, and it’s worth understanding how the SmartSync process differs before it’s actually required. This article walks through the full SmartSync failover and failback lifecycle, including what the commands do, why the sequence matters, and where the behavior diverges from what seasoned SyncIQ administrators will expect.
SmartSync replication is built on the Datamover service (isi_dm_d) and operates exclusively on Global Snapshots, referred to in its vernacular as ‘datasets’.
A dataset is an immutable, point-in-time view of the file system created via SnapshotIQ. Unlike a SyncIQ-replicated path, where the target is a live, writable mirror held read-only by convention, a SmartSync target path is backed by a series of versioned, lineage-aware datasets. This lineage is what makes efficient failback possible without re-baselining.
First, there are a few caveats to consider before getting into the workflow:
- Failover and failback are supported only by file-to-file replication (PowerScale cluster to PowerScale cluster). File-to-object cloud copy does not support either operation.
- The prerequisite state is a working repeat-copy relationship: a dataset creation policy running on the source and a repeat-copy policy replicating incrementally to the target. The baseline run must have completed successfully.
- Both clusters must have mutual TLS trust established and replication accounts configured in both directions. Trust is symmetric by design, which is what makes role reversal possible without reconfiguring certificates.
- The required privilege is ISI_PRIV_DATAMOVER. SmartSync also requires active SyncIQ and SnapshotIQ licenses on both clusters.
Workflow-wise, SmartSync failover and failback entails a seven-step process:
Steps 1 and 2 constitute the failover. Steps 3 through 5 constitute the failback. Steps 6 and 7 restore the original replication direction.
| Step | Action | Details |
| 1 | Failover (part i) | Make the target writeable; optionally revert to a clean dataset. |
| 2 | Failover (part ii) | Client writes occur on the target; create a dataset to capture changes. |
| 3 | Failback (part i) | Prepare the original source by reverting to the matching dataset. |
| 4 | Failback (part ii) | Perform the reverse incremental sync from target back to source. |
| 5 | Failback (part iii) | Make the original source writeable and redirect clients. |
| 6 | Reset (part i) | Prepare the original target for forward replication (revert and make read-only). |
| 7 | Reset (part ii) | Resume the original-direction repeat-copy policy. |
In practice, a planned failover may compress some steps. Similarly, an unplanned failover may skip others depending on the state of the source cluster.
Step 1: Making the Target Writeable (and Optionally Reverting)
Under normal repeat-copy operation, the target base path is read-only. SmartSync enforces this to prevent data written on the target from conflicting with incoming incremental updates from the source. The failover action unlocks the path with a single command:
target-1# isi dm setup-dir client-access --base-path /ifs/tgt Are you sure you would like to make base path /ifs/tgt writable? (yes/[no]): yes Created job: 1. Path will become writable.
In an unplanned failover where the source is simply unavailable, take the last replicated dataset and make it available for writes. However, in a ransomware or data-corruption scenario, the latest dataset on the target may itself be compromised. SmartSync addresses this with the ‘–dataset-revert’ option, which simultaneously makes the path writeable and rolls it back to a specific earlier dataset:
target-1# isi dm setup-dir client-access --base-path /ifs/tgt --dataset-revert 1 Are you sure you would like to revert base path /ifs/tgt to dataset ID 1? (yes/[no]): yes Created job: 1. Dataset will revert and path will become writable.
The dataset ID to revert to is identified beforehand using the ‘isi dm datasets list’ command, inspecting creation timestamps to find the last known-clean point in time. Once the revert completes, the target dataset that was reverted away from is removed from the dataset list, and the base path reflects the contents of the chosen dataset, ready for clients.
Step 2: Capturing Changes on the Failover Cluster
With clients now writing to the former target, a dataset creation policy is needed to capture those changes as Global Snapshots. This is necessary for Step 4 (failback sync) to have a reference point to sync from, and is created with a one-shot ‘run-now’ execution to immediately capture the current state, then configure recurrence for ongoing snapshots:
target-1# isi dm policies create fo_creation CREATION \ --priority NORMAL \ --run-now true \ --creation-account-id local \ --creation-base-path /ifs/tgt Created policy: 1.
At this point, if the original source is to be permanently decommissioned, the workflow stops here. The former target is now a fully independent source cluster. A new repeat-copy policy can be defined pointing at a fresh target, and normal forward replication resumes from this cluster with no re-baselining required anywhere downstream.
Step 3: Preparing the Original Source for Failback
Failback mandates that the original source be in a state the reverse-direction incremental sync can build upon. Specifically, the source must contain no datasets under the base path that post-date the dataset the target was using when it became writeable. If the source was still receiving new datasets from a running dataset creation policy at the moment of failover, those newer datasets must be removed before failback can proceed.
The ‘isi dm setup-dir replication-target’ CLI command handles this. It always requires a dataset revert and makes the base path read-only, transforming the cluster from an active source into a replication target ready to receive data:
source-1# isi dm setup-dir replication-target --base-path /ifs/src --dataset-revert 2 Are you sure you would like to revert base path /ifs/src to dataset 2? (yes/[no]): yes Created job: 1. Dataset will revert and path will become read-only
Dataset IDs are cluster-local, not globally synchronized, so the dataset ID on the source cluster will generally differ from the corresponding ID on the target. The preferred approach is to match by dataset revision and tree ID, not by the local integer ID. Running ‘isi dm dataset view <id>’ on both clusters and comparing the Dataset Revision and Dataset Global ID fields allows the match to be confirmed before the revert is issued.
Note that, if the final dataset creation policy on the source is configured with the ‘–creation-dataset-make-read-only true’ argument, the source base path is automatically made read-only as soon as that dataset is created. Combined with a scheduled dataset creation immediately before the planned maintenance window, this eliminates the need to manually run replication-target on the source, and ensures no writes land between the last sync and the failover.
Step 4: Reverse Incremental Sync
This is where SmartSync’s re-baselining elimination pays off in the failback scenario. A reverse-direction repeat-copy policy is created from the failover cluster (now acting as source) back to the original source (now acting as target). The ‘–reconnect true’ flag instructs SmartSync to resume from the last shared dataset rather than initiating a full baseline copy. Without it, the first run would transfer the entire dataset unnecessarily.
target-1# isi dm policies create failback_rc REPEAT_COPY \ --priority NORMAL \ --run-now true \ --repeat-copy-source-base-path /ifs/tgt \ --repeat-copy-base-source-account-id local \ --repeat-copy-base-target-account-id <source-cluster-account-id> \ --repeat-copy-base-target-base-path /ifs/src \ --reconnect true \ --repeat-copy-base-target-dataset-type FILE Created policy: 1025.
After the run completes, the job type can be verified in the ‘isi dm reports list’ command output. The type should be marked as ‘DATASET_INCREMENTAL’, confirming that SmartSync found the shared lineage and performed an incremental transfer rather than a full baseline. If it shows ‘DATASET_BASELINE’, the dataset revision match between the two clusters was incorrect and the revert (step 3) needs revisiting.
Step 5: Redirecting Clients Back to the Original Source
Once the failback sync has completed and the original source base path is current, the clients can be redirected. The ‘isi dm setup-dir client-access’ CLI command makes the path writeable without requiring a dataset revert, which is appropriate when the source is known-clean and the reverse sync is current:
source-1# isi dm setup-dir client-access --base-path /ifs/src Are you sure you would like to make base path /ifs/src writable? (yes/[no]): yes Created job: 1. Path will become writable.
Following this, any pertinent DNS entries, SmartConnect zone configuration, or NFS mount points can be updated as appropriate to redirect client traffic. The original source is now serving live client workloads again.
Step 6: Prepare the Original Target for Forward Sync
With clients back on the original source, the final pair of steps (steps 6 & 7) restores the normal A-to-B replication direction. First, the former failover cluster (B) is prep’d to become a replication target again. As such, any datasets created on it since the last successful reverse sync are reverted away, and the path is made read-only.
target-1# isi dm setup-dir replication-target \ --base-path /ifs/tgt \ --dataset-revert <latest-shared-dataset-id>
Step 7: Resume the Original Replication Direction
With the original target read-only and aligned to the same dataset revision as the source, the original forward repeat-copy policy is resumed with a manual ‘run-now’ trigger. SmartSync uses the shared lineage to perform an incremental update, again with no re-baseline:
source-1# isi dm policies modify <original-policy-id> --run-now true
Once this job completes successfully, the relationship is back to its pre-failover state: Source writeable, target read-only, incremental replication running on schedule. As such, the full cycle is complete.
PowerScale cluster admins who are familiar with SyncIQ will notice both familiar patterns plus notable differences when it comes to managing SmartSync failover. The following table summarizes the key operational distinctions:
| Aspect | SyncIQ | SmartSync |
| Failover command | isi sync policies modify –action allow-writes | isi dm setup-dir client-access |
| Source lock for failback | Implicit: resync-prep job prepares source as target | Explicit: isi dm setup-dir replication-target (always requires revert) |
| Re-baselining on failback | Not required (changelist-based incremental) | Not required (Global Snapshot lineage, –reconnect true) |
| Ransomware / data corruption recovery | Fail over to last clean sync snapshot via isi snapshot restore | Integrated: –dataset-revert in same failover command |
| Reverse sync direction | New SyncIQ policy, source becomes target | New REPEAT_COPY policy with –reconnect true from failover cluster |
| Restoring forward direction | Disable reverse policy; re-enable original policy | setup-dir replication-target on ex-target; modify original policy –run-now |
| Object / cloud replication | Not applicable (file only) | Cloud copy does not support failover or failback |
SmartSync provides two views into replication state during and after a failover or failback operation. Running jobs appear in ‘isi dm jobs list’; completed jobs move to ‘isi dm reports list’. For granular job detail, the ‘isi dm job view <id>’ CLI command provides per-task progress, and ‘isi_datamover_job_status <id>’ gives lower-level internal state. Transfer failures write a job-specific log to ‘/ifs/data/Isilon_Support/datamover/transfer_failures/<job-id>/’, and the primary log file is ‘/var/log/isi_dm.log’.
For RPO visibility across replication relationships, particularly useful in the lead-up to a planned failover, the ‘isi dm rpo reports list’ CLI command provides a per-policy view of how current the replicated data is relative to defined recovery objectives.
SmartSync’s support for pull policies, where the target cluster initiates and pulls data from the source, adds a useful dimension to DR topology design that SyncIQ does not offer. In a resource-constrained source cluster, configuring the target to pull rather than the source to push offloads the replication compute burden to the DR site, leaving primary client I/O unaffected.
For the failback reverse sync specifically (Step 4), the policy is always issued from the cluster that holds the current writeable data, pushing to the recovering original source. There is no functional difference in the failover/failback sequence between clusters that used push versus pull in normal operations; both directions use the same commands. The key is that account IDs and base paths are swapped to reflect the reversed roles.
SmartSync failover and failback is a structured, seven-step workflow anchored on Global Snapshot lineage and three core commands: ‘isi dm setup-dir client-access’ (unlock a path for writes), ‘isi dm setup-dir replication-target’ (lock a path as a replication target and revert to a known state), and a reverse ‘REPEAT_COPY’ policy with ‘–reconnect true’ (incremental failback without re-baselining). The sequence handles both unplanned outage (fail over to last clean dataset, write locally, sync back) and planned maintenance (using the ‘–creation-dataset-make-read-only’ flag to cleanly quiesce the source before cutover).
Note the following important differences from SyncIQ: There is no equivalent of the ‘resync-prep job’, The ‘isi dm setup-dir replication-target’ CLI command, and its mandatory dataset revert, serve that function. Dataset ID matching between clusters is by revision and global ID, not by local integer. And –reconnect true on the reverse repeat-copy policy is non-negotiable for incremental failback.
Note also that SmartSync failover and failback is only supported for file-to-file replication. File-to-object cloud copy (replication to ECS, AWS, Azure, or GCP) does not support these operations and is intended for backup and archival workflows only. Verify the Job Type field in ‘isi dm reports list’ after a reconnect-mode failback sync: It must read ‘DATASET_INCREMENTAL’ to confirm that no re-baseline occurred.