Author: Nick Trimbee

The previous article in this series looked at the OneFS Multi-party Authorization architecture, which requires that one or more additional trusted parties sign off on requested changes for certain privileged actions within a PowerScale cluster.

At a high-level, the MPA workflow is as follows:

1. The first step is registration, which requires two users with the approval privilege, both registered as internal approvers. And note that merely having an RBAC approval privilege is not enough – there is a secure process to register the users. Once there are two registered approval users, then it’s on to step #2, where the approval admin can enable the MPA feature.
2. Once MPA is enabled, any of the predefined ‘privileged actions’ will require explicit approval from internal approvers.
3. Step three involves generating a Privileged Action Approval Request (PAA) – either manually, or automatically when a privileged action is attempted.
4. Next, one of the registered MPA approvers blesses or rejects the PAA Request.
5. And finally, assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

In order to approve requests, an approver needs to complete a secure one-time registration by entering the six-digit TOTP code, etc.

MPA can be configured and managed via the OneFS CLI, platform API, or WebUI. OneFS 9.12 sees the addition of a new ‘isi mpa’ CLI command set who’s basic syntax is as follows:

# isi mpa
Description:
    Manage Multi-Party Authorization.

Usage:
    isi mpa {<action> | <subcommand>}
        [--timeout <integer>]
        [{--help | -h}]

Actions:
    complete-registration    Complete Multi-Party Authorization registration.
    initiate-registration    Initiate Multi-Party Authorization registration.

Subcommands:
    approvers                Manage Multi-Party Authorization approvers.
    ca-certificates          Manage CA-Certificates for Multi-Party
                             Authorization.
    requests                 Manage Multi-Party Authorization requests.
    settings                 Manage Multi-Party Authorization settings.

For example, the CLI syntax to view the MPA global settings:

# isi mpa settings global view

Multi Party Authorization is enabled

Similarly, the platform API MPA endpoints reside under ‘/platform/23/mpa’. For example, the endpoint to get the MPA global settings:

https://<cluster_ip_addr>:8080/platform/23/mpa/settings/global

{

"last_update_time" : 1753132456,

"mpa_enabled" : true

}

The WebUI configuration and management portal for MPA is located at Access > Multi-Party Authorization. For example, navigating to Access > Multi-Party Authorization will yield the MPA global settings:

The CLI commands and platform API endpoints for registration, query, and approval are as follows:

The five basic steps for the MPA configuration and management are as follows:

1. First, Create users and grant approval privilege.

The approver registration and enablement workflow is as follows:

Approver registration be configured from the CLI with the following set of commands.

First, configure two approver users – in this case ‘mpa-approver1’ and ‘mpa-approver2’:

# isi auth users create mpa-approver1 --enabled true --password <passwd>

# isi auth users create mpa-approver2 --enabled true --password <passwd>

Next, create a group for these users. Eg. ‘mpa-group’:

# isi auth group create mpa-group

Add the approver users to their new group:

# isi auth group modify mpa-group --add-user mpa-approver1

# isi auth group modify mpa-group --add-user mpa-approver2

Then add the new group (mpa-group) to the RBAC ApprovalAdmin role:

# isi –-debug auth roles modify ApprovalAdmin --add-group mpa-group

If needed, configure approvers’ SSH login to the cluster.

# isi auth roles create ssh-role

# isi auth roles modify ssh-role --add-group mpa-group --add-priv-read ISI_PRIV_LOGIN_SSH

Next, add the approver users to MPA and complete their registration. This involves the approver user logging in to their account and initiating registration, using the secret or embedded URL in conjunction with a third party code generator to create a time-based one-time password or TOTP.

Here, the ‘mpa-approver1’ user is registering first:

# isi mpa initiate-registration

   Initiated registration successfully: (Online utility can be used to   convert URL to QR code for use with authenticator)

   account:  Dell Technologies

   algorithm:  SHA1

   digits:  6

   issuer:  mpa-approver1

   period:  30

   secret:  5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI

   url:  otpath://totp/Dell%Technologies:mpa-approver1?secret=5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI&issuer=Dell%20Technologies&algorithm=SHA1&digits=6&period=30

The user secret or URL returned can be used to set up a TOTP in Google Authenticator, or another similar application.

Specifically, this secure one-time registration involves:

a. First, initiating the OneFS registration process to obtain a URI, which can then be generated into a QR code by a 3^rd party app if desired. For example:

b. Next, converting the QR code or URI into a time-based one-time password code via Google Authenticator, Microsoft Authenticator, or another TOTP app:

c. Finally, completing the OneFS approver registration wizard:

Or from the CLI:

# isi mpa complete-registration

totp_code:  ******

Approver registered.

This process should then be repeated for the second (ie. ‘mpa-approver2’) role.

2. Once the approvers are registered, MPA can now be enabled via its global settings.

MPA can be enabled from the WebUI under Access > Multi-Party Authorization > Settings:

Note that MPA cannot be enabled on clusters without having at least two registered approvers:

When successfully enabled, the following status banner will be displayed: 

Alternatively, enabling via the CLI can be done with the following command syntax:

# isi mpa settings global modify –enable true

# isi mpa settings global view

Multi Party Authorization is enabled

3. Now MPA is up and running, privileged action requests can be generated.

We’ll use the new OneFS 9.12 Secure Snapshots feature as an example.

Once MPA is enabled on a cluster, any requests to execute a privileged request are paused pending approval.

In this case, an attempt to delete a secure snapshot results in a warning, and the requested action is suspended until approved by an authorizing party.

4. Request Approval.

The privileged action requests can now be approved or rejected, as appropriate.

To private consent for the operation, the authorizing administrator clicks the ‘Approve’ button for the pending request, located under Access > Multi-Party Authorization > Requests:

The authorizing party is prompted for their time-based one-time password (TOTP) security authentication code as part of the approval process:

Or via the CLI:

# isi mpa requests list

# isi mpa requests approve <id> <comment> <approved> --totp-code <******> --approval-valid-before <timestamp> --zone <zone>

5. Finally, the privileged action is ready to be executed.

Once approval has been granted, the WebUI reports a successful approval status.

The privileged secure snapshot delete operation is now able to proceed as expected:

Or from the CLI:

# isi mpa requests view <id>

In the next and final article in this series, we’ll turn our attention to MPA management and troubleshooting.

As we saw in the previous article in this series, OneFS Multi-party Authorization (MPA) requires that one or more additional trusted parties sign off on requested changes for certain privileged actions within a PowerScale cluster.

A prerequisite to enabling MPA in OneFS 9.12 is the requirement for two users with the approval privilege, both registered as internal approvers. Once MPA has been activated on a cluster, any of the predefined ‘privileged actions’ will require explicit approval. In order for a privileged action, such as secure snapshot deletion, to complete, one of the registered MPA approvers must bless the PAA request. Finally, assuming approval was granted, the privileged action can be executed.

The MPA privileged action approval workflow can be summarized in the following request state diagram:

Under the hood, MPA request lifecycle configuration data is stored in the form of key-value pairs, and the MPA Approvers registration status, PAA requests, and request approval details are kept in a shared SQLite database.

Note that MPA approver users can only reside in the System Zone and require the new ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST privileges at a minimum.

MPA includes privileged actions, which extend across the S3 protocol, supporting immutable bucket retention and logging, cluster hardening, and the main one which is the new secure snapshots functionality.

As such, the full complement of these privileged actions in OneFS 9.12 is as follows:

A number of new RBAC privileges are introduced in 9.12 to support MPA, including MPA request, MPA approval, which allows an approver to bless or reject a privileged action request, plus MPA upload, global and access zone settings:

There’s also the new ‘Approval Admin’ default role, and a number of other roles that see the addition of MPA privileges, including SecurityAdmin, SystemAdmin, zone admin, and basic user, etc.

The ApprovalAdmin role is automatically assigned all the required privileges for MPA approval and configuration. For example:

# isi auth roles list | grep -i appr

ApprovalAdmin

# isi auth roles view ApprovalAdmin
       Name: ApprovalAdmin
Description: Allows MPA request approval.
    Members: root
             admin
 Privileges
             ID: ISI_PRIV_LOGIN_PAPI
     Permission: +

             ID: ISI_PRIV_MPA_APPROVAL
     Permission: w

             ID: ISI_PRIV_MPA_REQUEST
     Permission: w

             ID: ISI_PRIV_MPA_SETTINGS_GLOBAL
     Permission: r

             ID: ISI_PRIV_MPA_SETTINGS_ZONE
     Permission: r

The minimum MPA privileges required are ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST.

Here are the system access zone roles that see the addition of MPA privileges in OneFS 9.12:

Similarly, the OneFS 9.12 MPA privileges for non-system zone roles:

MPA approvers can use any of a cluster’s authentication providers, such as file, local, Active Directory or LDAP, and can they grant or reject any MPA request from any zone. However, an approver cannot approve their own request, regardless of whether it’s created by them or created for them.

The CLI commands and platform API endpoints for registration, query, and approval are as follows:

Note that the approver user can easily re-register at any time, say if their TOTP is lost, or for another security concern.

An approval request can be in one of five states: Approved, Cancelled, Completed, Pending, or Rejected.

These states, and their various expiry times, can be viewed with the following CLI command syntax:

# isi mpa settings request-lifecycle list

Approved
    Description: expire in configured days for privilege action execution.
    Expire_time: 7d

Cancelled
    Description: remove from system in configured days.
    Expire_time: 7d

Completed
    Description: remove from system in configured days.
    Expire_time: 30d

Pending
    Description: expire in configured days can be updated by user or approver.
    Expire_time: 7d

Rejected

    Description: remove from system in configured days.
    Expire_time: 30d

The next article in this series will focus on the specific configuration, enablement, and approval procedures for MPA.

Among a handful of new security features, OneFS 9.12 introduces Multi-Party Authorization (MPA). MPA is an administrative approval mechanism that requires at least one additional trusted party to sign off on a requested change, for certain privileged actions within a PowerScale cluster.

As such, MPA helps to mitigate the risk of data loss or system configuration damage from critical actions, by vastly reducing the likelihood of accidental or malicious execution of consequential operations. Conceptually, MPA is fairly analogous to a bank’s safe deposit box, which requires dual control: Both the authorized owner and a bank official must be present in order to open the vault and unlock the box, ensuring that no single party can access its contents independently (dual authorization). MPA enforces a similar security control mechanism wherein operations involving critical or sensitive systems or data require explicit approval from multiple authorized entities. This ensures that no single actor can execute high-impact actions unilaterally, thereby mitigating risk and enhancing operational integrity through enforced oversight.

As such, many enterprises require MPA in order to meet industry data protection requirements, in addition to internal security mandates and best practices.

MPA can be configured and managed from the CLI, WebUI or platform API, but note that the feature is not activated by default. Once MPA has been enabled and configured, all predefined ’privileged actions’ require additional approval from an authorizing user. So this is markedly different from earlier OneFS releases, where the original requester would likely have had the basic rights and been able to execute that action in its entirety themselves.

MPA provides an approval mechanism for a set of ‘privileged actions’, which are defined in OneFS. Additionally, a new default ‘approval admin’ RBAC role is added in OneFS 9.12 specifically for this purpose. This ApprovalAdmin role is automatically assigned all the required privileges for MPA approval and configuration. Specifically:

There are no special licensing requirements for enabling or running MPA. So, upon installation or upgrade to OneFS 9.12, once two or more approval users have been registered, the MPA feature itself can be enabled:

Note that MPA is incompatible with, and therefore not supported on, a PowerScale cluster that is already running in Compliance Mode.

Once MPA is up and running, any approval that’s granted for a privileged operation will remain valid and viable for a defined duration. When approving a request, the approver can specify an expiry time. Once that allotted time has passed, the approved request can no longer be used to execute an action, and a new approval must be generated. If the approver does not specify an expiry time, then, by default, their approval will be valid for seven days.

At a high-level, the MPA workflow is as follows:

1. The first step is registration, which requires two users with the approval privilege, both registered as internal approvers. And note that merely having an RBAC approval privilege is not enough – there is a secure process to register the users. Once there are two registered approval users, the approval admin can enable the MPA feature.

   With MPA is enabled, Privileged Action Approval Requests (PAA) can be generated – either manually, or automatically when a privileged action is attempted. Any of the predefined ‘privileged actions’ will require explicit approval from internal approvers. Next, one of the registered MPA approvers blesses or rejects the PAA Request. And finally, assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

   Step	Action	Description
   1	Approver Registration	Registration requires two users with the approval privilege, both registered as internal approvers. MPA requires a secure process to register the users.
   2	MPA Enablement	Enable MPA via the CLI, WebUI, or platform API. Any of the predefined ‘privileged actions’ will require explicit approval from internal approvers.
   3	Request Creation	Generate a Privileged Action Approval Request (PAA) – either manually, or automatically when a privileged action is attempted.
   4	Request Approval	One of the registered MPA approvers blesses or rejects the PAA request.
   5	Request Execution	Assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

MPA approver users can only reside in the System Zone, and require the new ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST privileges at a minimum. So while a request can be generated from any zone, an approver for that request must reside in the system zone.

An approval request can be in one of the following states:

Note that, in order to approve or reject requests, an approver needs to complete a secure one-time registration.

In OneFS 9.12, the MPA request lifecycle configuration is predefined, and no updates are supported. Only the Approval API (approval_valid_before) can shorten MPA request expiration time in approved status. MPA requests are zone scoped and managed by MPA request lifecycle configuration. An authorized user can view or update a request while it’s in a ‘pending’ state, but only an MPA approver user can move the request to an ‘approved’ or ‘rejected’ state. While an approval is valid (i.e. not expired), the original requester can use it to execute the privileged action. A request stays in an ‘approved’ state until its approval expires, at which point it is moved to a ‘completed’ state.

MPA includes privileged actions, which extend across the S3 protocol, supporting immutable bucket retention and logging, cluster hardening, and the main one which is the new secure snapshots functionality. These are available for selection in the MPA configuration, for example from the WebUI MPA Requests dropdown menu:

In the next article in this series, we’ll look closer at the architectural fundaments that constitute MPA.

Dell PowerScale is already powering up the summer with the launch of the innovative OneFS 9.12 release, which shipped today (14^th August 2025). This new 9.12 release has something for everyone, introducing PowerScale innovations in security, serviceability, reliability, protocols, and ease of use.

OneFS 9.12 represents the latest version of PowerScale’s common software platform for on-premises and cloud deployments. This can make it a excellent choice for traditional file shares and home directories, vertical workloads like M&E, healthcare, life sciences, financial services, plus generative and agentic AI, and other ML/DL and analytics applications.

PowerScale’s scale-out architecture can be deployed on-site, in co-lo facilities, or as customer-managed PowerScale for Amazon AWS and Microsoft Azure deployments, providing core to edge to cloud flexibility, plus the scale and performance needed to run a variety of unstructured workflows on-prem or in the public cloud.

With data security, detection, and monitoring being paramount in this era of unprecedented cyber threats, OneFS 9.12 brings an array of new features and functionality to keep your unstructured data and workloads more available, manageable, and secure than ever.

Protocols

On the S3 object protocol front, OneFS 9.12 sees the debut of new security and immutability functionality. S3 Object Lock extends the standard AWS S3 Object Lock model with PowerScale’s own ‘Bucket-Lock’ protection mode semantics. Object Lock capabilities can operate on a per-zone basis and per-bucket, using the cluster’s compliance clock for the date and time evaluation of object’s retention. Additionally, S3 protocol access logging and bucket logging are also enhanced in this new 9.12 release.

Networking

As part of PowerScale’s seamless protocol failover experience for customers, OneFS 9.12 sees SmartConnect’s default IP allocation method for new pools move to ‘dynamic’. While SMB2 and SMB3 are the primary focus, all protocols benefit from this enhancement, including SMB, NFS, S3, and HDFS. Legacy pools will remain unchanged upon upgrade to 9.12, but any new pools will automatically be provisioned as dynamic (unless manually configured as ‘static’).

Security

In the interests of increased security and ransomware protection, OneFS 9.12 includes new Secure Snapshots functionality. Secure Snapshots provide true snapshot immutability, as well as protection for snapshot schedules, in order to protect against alteration or deletion, either accidentally or by a malicious actor.

Secure snapshots are built upon Multi-party Authorization (MPA), also introduced in OneFS 9.12. MPA prevents an individual administrator from executing privileged operations, such as configuration changes on snapshots and snapshot schedules, by requiring two or more trusted parties to sign off on a requested change for the privileged actions within a PowerScale cluster.

OneFS 9.12 also introduces support for common access cards (CAC) and personal identity verification (PIV) smart cards, providing physical multi-factor authentication (MFA), allowing users to SSH to a PowerScale cluster using the same security badge that grants them access into their office. In addition to US Federal mandates, CAC/PIV integration is a requirement for many security conscious organizations across the public and private sectors.

Upgrade

One-click upgrades in OneFS 9.12 allow a cluster to automatically display and download available  trusted upgrade packages from Dell Support, which can be easily applied via ‘one click installation’ from the OneFS WebUI or CLI. Upgrade package versions are automatically managed by Dell in accordance with a cluster’s telemetry data.

Support

OneFS 9.12 introduces an auto-healing capability, where the cluster detects problems using the HealthCheck framework and automatically executes a repair action for known issues and failures. This helps to increase cluster availability and durability, while reducing the time to resolution and the need for technical support engagements. Furthermore, additional repair-actions can be added at any point, outside of the general OneFS release cycle.

Hardware Innovation

On the platform hardware front, OneFS 9.12 also introduces an HDR Infiniband front-end connectivity option for the PowerScale PA110 performance and backup accelerator. Plus, 9.12 also brings a fast reboot enhancement to the high-memory PowerScale F-series nodes.

In summary, OneFS 9.12 brings the following new features and functionality to the Dell PowerScale ecosystem:

We’ll be taking a deeper look at OneFS 9.12’s new features and functionality in future blog articles over the course of the next few weeks.

Meanwhile, the new OneFS 9.12 code is available on the Dell Support site, as both an upgrade and reimage file, allowing both installation and upgrade of this new release.

For existing clusters running a prior OneFS release, the recommendation is to open a Service Request with to schedule an upgrade. To provide a consistent and positive upgrade experience, Dell is offering assisted upgrades to OneFS 9.12 at no cost to customers with a valid support contract. Please refer to Knowledge Base article KB544296 for additional information on how to initiate the upgrade process.

As we saw in the previous articles in this series, SmartSync in OneFS 9.11 enjoys the addition of backup-to-object functionality, which delivers high performance, full-fidelity incremental replication to ECS, ObjectScale, Wasabi, and AWS S3 & Glacier IR object stores.

This new SmartSync backup-to-object functionality supports the full spectrum of OneFS path lengths, encodings, and file sizes up to 16TB – plus special files and alternate data streams (ADS), symlinks and hardlinks, sparse regions, and POSIX and SMB attributes.

In addition to the standard ‘isi dm’ command set, the following CLI utility can also come in handy for tasks such as verifying the dataset ID for restoration, etc:

# isi_dm browse

For example, to query the SmartSync accounts and datasets:

# isi_dm browse

<no account>:<no dataset> $ list-accounts

000000000000000100000000000000000000000000000000 (tme-tgt)

ec2a72330e825f1b7e68eb2352bfb09fea4f000000000000 (DM Local Account)

fd0000000000000000000000000000000000000000000000 (DM Loopback Account)

<no account>:<no dataset> $ connect-account 000000000000000100000000000000000000000000000000

tme-tgt:<no dataset> $ list-datasets

1       2025-07-22T10:23:33+0000        /ifs/data/zone3

2       2025-07-22T10:23:33+0000        /ifs/data/zone4

1025    2025-07-22T10:25:01+0000        /ifs/data/zone3

2049    2025-07-22T10:30:04+0000        /ifs/data/zone4

tme-tgt:<no dataset> $ connect-dataset 2

tme-tgt:2 </ifs/data/zone4:> $ ls

home                           [dir]

zone2_sync1753179349           [dir]

tme-tgt:2 </ifs/data/zone4:> $ cd zone2_sync1753179349

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ ls

home                           [dir]

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $

Or for additional detail:

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ settings output-to-file-on /tmp/out.txt

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ settings verbose-on

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ list-datasets

1       2025-07-22T10:23:33+0000        /ifs/data/zone3 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=1 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=1 } }

2       2025-07-22T10:23:33+0000        /ifs/data/zone4 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=2 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=2 } }

1025    2025-07-22T10:25:01+0000        /ifs/data/zone3 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=1 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=3 } }

2049    2025-07-22T10:30:04+0000        /ifs/data/zone4 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=2 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=4 } }

But when it comes to monitoring and troubleshooting SmartSync, there are a variety of diagnostic tools available. These include:

SmartSync info and errors are typically written to /var/log/isi_dm.log and /var/log/messages, while DM jobs transfer failures generate a log specific to the job ID under /ifs/data/Isilon_Support/datamover/transfer_failures.

Once a policy is running, the job status is reported via ‘isi dm jobs list’. Once complete, job histories are available by running ‘isi dm historical jobs list’. More details for a specific job can be gleaned from the ‘isi dm job view’ command, using the pertinent job ID from the list output above. Additionally, the ‘isi_datamover_job_status’ command with the job ID as an argument will also supply detailed information about a specific job.

Once running, a DM job can be further controlled via the ‘isi dm jobs modify’ command, and available actions include cancel, partial-completion, pause, or resume.

If a certificate authority (CA) is not correctly configured on a PowerScale cluster, the SmartSync daemon will not start, even though accounts and policies can still be configured. Be aware that the failed policies will not be reported via ‘isi dm jobs list’ or ‘isi dm historical-jobs list’ since they never started. Instead, an improperly configured CA is reported in the /var/log/isi_dm.log as follows:

Certificates not correctly installed, Data Mover service sleeping: At least one CA must be installed: No such file or directory from dm_load_certs_from_store (/b/mnt/src/isilon/lib/isi_dm/isi_dm_remote/src/rpc/dm_tls.cpp:197 ) from dm_tls_init (/b/mnt/src/isilon/lib/isi_dm/isi_dm_remote/src/rpc/dm_tls.cpp:279 ): Unable to load certificate information

Once a CA and identity are correctly configured, the SmartSync service automatically activates. Next, SmartSync attempts a handshake with the target. If the CA or identity is mis-configured, the handshake process fails, and generates an entry in /var/log/isi_dm.log. For example:

2025-07-30T12:38:17.864181+00:00 GEN-HOP-NOCL-RR-1(id1) isi_dm_d[52758]: [0x828c0a110]: /b/mnt/src/isilon/lib/isi_dm/isi_dm_remote/src/acct_mon.cpp:dm_acc tmon_try_ping:348: [Fiber 3778] ping for account guid: 0000000000000000c4000000000000000000000000000000, result: dead

Note that the full handshake error detail is logged if the SmartSync service (isi_dm_d) is set to log at the ‘info’ or ‘debug’ level using isi_ilog:

# isi_ilog -a isi_dm_d --level info+

Valid ilog levels include:

fatal error err notice info debug trace

error+ err+ notice+ info+ debug+ trace+

A copy or repeat-copy policy requires an available dataset for replication before running. If a dataset has not been successfully created prior to the copy or repeat-copy policy job starting for the same base path, the job is paused. In the following example, the base path of the copy policy is not the same as that of the dataset policy, hence the job fails with a “path doesn’t match…” error.

# ls -l /ifs/data/Isilon_support/Datamover/transfer_failures

Total 9

-rw-rw----   1 root  wheel  679  July 20 10:56 baseline_failure_10

# cat /ifs/data/Isilon_support/Datamover/transfer_failures/baseline_failure_10

Task_id=0x00000000000000ce, task_type=root task ds base copy, task_state=failed-fatal path doesn’t match dataset base path: ‘/ifs/test’ != /ifs/data/repeat-copy’:

from bc_task)initialize_dsh (/b/mnt/src/isilon/lib/isi_dm/isi_dm/src/ds_base_copy

from dmt_execute (/b/mnt/src/isilon/lib/isi_dm/isi_dm/src/ds_base_copy_root_task

from dm_txn_execute_internal (/b/mnt/src/isilon/lib/isi_dm/isi_dm_base/src/txn.cp

from dm_txn_execute (/b/mnt/src/isilon/lib/isi_dm/isi_dm_base/src/txn.cpp:2274)

from dmp_task_spark_execute (/b/mnt/src/isilon/lib/isi_dm/isi_dm/src/task_runner.

Once any errors for a policy have been resolved, the ‘isi dm jobs modify’ command can be used to resume the job.

As we saw in the previous article in this series, SmartSync in OneFS 9.11 sees the addition of backup-to-object, which provides high performance, full-fidelity incremental replication to ECS, ObjectScale, Wasabi, and AWS S3 & Glacier IR object stores.

This new SmartSync backup-to-object functionality supports the full spectrum of OneFS path lengths, encodings, and file sizes up to 16TB – plus special files and alternate data streams (ADS), symlinks and hardlinks, sparse regions, and POSIX and SMB attributes. Specifically:

SmartSync backup-to-object operates on user-defined data set, which are essentially OneFS file system snapshots with plus additional properties.

A data set creation policy takes snapshots and creates a data set out of it. Additionally, there are also copy and repeat copy policies which are the policies that will transfer that data set to another system. And the execution of these two policy types can be linked and scheduled separately. So you can have one schedule for your data set creation, say to create a data set every hour on a particular path. And you can have a tiered or different distribution system for the actual copy itself. For example, to copy every hour to a hot DR cluster in data center A. But also copy every month to a deep archive cluster in data center B. So all these things are possible now, without increasing the bloat of snapshots on the system, since they’re now able to be shared.

Currently, SmartSync does not have a WebUI presence, so all its configuration is either via the command-line or platform API.

Here’s the procedure for crafting a baseline replication config:

Essentially, create the replication account, which in OneFS 9.11 will be either Dell ECS or Amazon AWS. Then configure that dataset creation policy, run it, and, if desired, create a repeat-copy policy. These specific steps with their CLI syntax include:

1. Create a replication account:

# isi dm account create --account-type [AWS_S3 | ECS_S3]

2. Configure a dataset creation policy

# isi dm policies create [Policy Name] --policy-type CREATION

3. Run the dataset creation policy:

# isi dm policies list

# isi dm policies modify [Creation policy id] –-run-now=true

# isi dm jobs list

# isi dm datasets list

4. create a repeat-copy policy

# isi dm policies create [Policy Name] --policy-type=' REPEAT_COPY'

5. Run the repeat-copy policy:

# isi dm policies list

# isi dm policies modify [Repeat-copy policy id] –-run-now=true

6. View the data replication job status

# isi dm jobs list

Similarly for an incremental replication config:

Note that the dataset creation policy and repeat-copy policy are already created in the baseline replication configure and can be ignored.

Incremental replication using the dataset create and repeat-copy policies from the previous slide’s baseline config.

1. Run the dataset creation policy

# isi dm policies list

# isi dm policies modify [Creation policy id] –-run-now=true

# isi dm jobs list

# isi dm datasets list

2. Run the repeat-copy policy:

# isi dm policies list

# isi dm policies modify [Repeat-copy policy id] –-run-now=true

3. View the data replication incremental job status

# isi dm jobs list

And here’s the basic procedure for creating and running a partial or full restore:

Note that the replication account is already created on the original cluster and the creation step can be ignored.  Replication account creation is only required if restoring the dataset to a new cluster.

Additionally, partial restoration involves a subset of the directory structure, specified via the ‘source path’ , whereas full restoration invokes a restore of the entire dataset.

The process includes creating the replication account if needed, finding the ID of the dataset to be restored, creating and running the partial or full restoration policy, and checking the job status to verify it ran successfully.

1. Create a replication account:

# isi dm account create --account-type [AWS_S3 | ECS_S3]

For example:

# isi dm account create --account-type ECS_S3 --name [Account Name] --access-id [access-id] --uri [URI with bucket-name] --auth-mode CLOUD --secret-key [secret-key] --storage-class=[For AWS_S3 only: STANDARD or GLACIER_IR]

2. Verify the dataset ID for restoration:

# isi_dm browse

Checking the following attributes:

• list-accounts
• connect-account [Source Account ID created in step 1]
• list-datasets
• connect-dataset [Dataset id]

3. Create a partial or full restoration policy

# isi dm policies create [Policy Name] --policy-type='COPY'

4. Run the partial or full restoration policy:

# isi dm policies modify [Restoration policy id] –-run-now=true

5. View the data restoration job status

# isi dm jobs list

OneFS 9.11 also introduces recovery point objective or RPO alerts for SmartSync, but note that these are for repeat-copy policies only. These RPO alerts can be configured through the replication policy by adding the desired time value to the ‘repeat-copy-rpo-alert’ parameter. If this configured threshold is exceeded, an RPO alert is triggered. This RPO alert is automatically resolved after the next successful policy job run.

Also be aware that the default time value for a repeat copy RPO is zero, which instructs SmartSync to not generate RPO alerts for that policy.

The following CLI syntax can be used to create a replication policy, with the ‘–repeat-copy-rpo-alert’ flag set for the desired time:

# isi dm policies create [Policy Name] --policy-type=' REPEAT_COPY' --enabled='true' --priority='NORMAL' --repeat-copy-source-base-path=[Source Path] --repeat-copy-base-base-account-id=[Source account id] --repeat-copy-base-source-account-id=[Source account id] --repeat-copy-base-target-account-id=[Target account id] --repeat-copy-base-new-tasks-account=[Source account id] --repeat-copy-base-target-dataset-type='FILE_ON_OBJECT_BACKUP' --repeat-copy-base-target-base-path=[Bucket Name] --repeat-copy-rpo-alert=[time]

And similarly to change the RPO alert configuration on an existing replication policy:

# isi dm policies modify [Policy id] --repeat-copy-rpo-alert=[time]

An alert is triggered and corresponding CELOG event created if the specified RPO for the policy is exceeded. For example:

# isi event list

ID   Started     Ended       Causes Short                     Lnn  Events  Severity

--------------------------------------------------------------------------------------

1898 07/15 00:00 07/15 00:00 SW_CELOG_HEARTBEAT               1    1       information

2012 07/15 06:03 --          SW_DM_RPO_EXCEEDED               2    1       warning

--------------------------------------------------------------------------------------

And then once RPO alert has been resolved after a successful replication policy job run:

# isi event list

ID   Started     Ended       Causes Short                     Lnn  Events  Severity

--------------------------------------------------------------------------------------

1898 07/15 00:00 07/15 00:00 SW_CELOG_HEARTBEAT               1    1       information

2012 07/15 06:03 07/15 06:12 SW_DM_RPO_EXCEEDED               2    2       warning

--------------------------------------------------------------------------------------