Tag: Isilon

The previous article in this series looked at the OneFS Multi-party Authorization architecture, which requires that one or more additional trusted parties sign off on requested changes for certain privileged actions within a PowerScale cluster.

At a high-level, the MPA workflow is as follows:

1. The first step is registration, which requires two users with the approval privilege, both registered as internal approvers. And note that merely having an RBAC approval privilege is not enough – there is a secure process to register the users. Once there are two registered approval users, then it’s on to step #2, where the approval admin can enable the MPA feature.
2. Once MPA is enabled, any of the predefined ‘privileged actions’ will require explicit approval from internal approvers.
3. Step three involves generating a Privileged Action Approval Request (PAA) – either manually, or automatically when a privileged action is attempted.
4. Next, one of the registered MPA approvers blesses or rejects the PAA Request.
5. And finally, assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

In order to approve requests, an approver needs to complete a secure one-time registration by entering the six-digit TOTP code, etc.

MPA can be configured and managed via the OneFS CLI, platform API, or WebUI. OneFS 9.12 sees the addition of a new ‘isi mpa’ CLI command set who’s basic syntax is as follows:

# isi mpa
Description:
    Manage Multi-Party Authorization.

Usage:
    isi mpa {<action> | <subcommand>}
        [--timeout <integer>]
        [{--help | -h}]

Actions:
    complete-registration    Complete Multi-Party Authorization registration.
    initiate-registration    Initiate Multi-Party Authorization registration.

Subcommands:
    approvers                Manage Multi-Party Authorization approvers.
    ca-certificates          Manage CA-Certificates for Multi-Party
                             Authorization.
    requests                 Manage Multi-Party Authorization requests.
    settings                 Manage Multi-Party Authorization settings.

For example, the CLI syntax to view the MPA global settings:

# isi mpa settings global view

Multi Party Authorization is enabled

Similarly, the platform API MPA endpoints reside under ‘/platform/23/mpa’. For example, the endpoint to get the MPA global settings:

https://<cluster_ip_addr>:8080/platform/23/mpa/settings/global

{

"last_update_time" : 1753132456,

"mpa_enabled" : true

}

The WebUI configuration and management portal for MPA is located at Access > Multi-Party Authorization. For example, navigating to Access > Multi-Party Authorization will yield the MPA global settings:

The CLI commands and platform API endpoints for registration, query, and approval are as follows:

The five basic steps for the MPA configuration and management are as follows:

1. First, Create users and grant approval privilege.

The approver registration and enablement workflow is as follows:

Approver registration be configured from the CLI with the following set of commands.

First, configure two approver users – in this case ‘mpa-approver1’ and ‘mpa-approver2’:

# isi auth users create mpa-approver1 --enabled true --password <passwd>

# isi auth users create mpa-approver2 --enabled true --password <passwd>

Next, create a group for these users. Eg. ‘mpa-group’:

# isi auth group create mpa-group

Add the approver users to their new group:

# isi auth group modify mpa-group --add-user mpa-approver1

# isi auth group modify mpa-group --add-user mpa-approver2

Then add the new group (mpa-group) to the RBAC ApprovalAdmin role:

# isi –-debug auth roles modify ApprovalAdmin --add-group mpa-group

If needed, configure approvers’ SSH login to the cluster.

# isi auth roles create ssh-role

# isi auth roles modify ssh-role --add-group mpa-group --add-priv-read ISI_PRIV_LOGIN_SSH

Next, add the approver users to MPA and complete their registration. This involves the approver user logging in to their account and initiating registration, using the secret or embedded URL in conjunction with a third party code generator to create a time-based one-time password or TOTP.

Here, the ‘mpa-approver1’ user is registering first:

# isi mpa initiate-registration

   Initiated registration successfully: (Online utility can be used to   convert URL to QR code for use with authenticator)

   account:  Dell Technologies

   algorithm:  SHA1

   digits:  6

   issuer:  mpa-approver1

   period:  30

   secret:  5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI

   url:  otpath://totp/Dell%Technologies:mpa-approver1?secret=5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI&issuer=Dell%20Technologies&algorithm=SHA1&digits=6&period=30

The user secret or URL returned can be used to set up a TOTP in Google Authenticator, or another similar application.

Specifically, this secure one-time registration involves:

a. First, initiating the OneFS registration process to obtain a URI, which can then be generated into a QR code by a 3^rd party app if desired. For example:

b. Next, converting the QR code or URI into a time-based one-time password code via Google Authenticator, Microsoft Authenticator, or another TOTP app:

c. Finally, completing the OneFS approver registration wizard:

Or from the CLI:

# isi mpa complete-registration

totp_code:  ******

Approver registered.

This process should then be repeated for the second (ie. ‘mpa-approver2’) role.

2. Once the approvers are registered, MPA can now be enabled via its global settings.

MPA can be enabled from the WebUI under Access > Multi-Party Authorization > Settings:

Note that MPA cannot be enabled on clusters without having at least two registered approvers:

When successfully enabled, the following status banner will be displayed: 

Alternatively, enabling via the CLI can be done with the following command syntax:

# isi mpa settings global modify –enable true

# isi mpa settings global view

Multi Party Authorization is enabled

3. Now MPA is up and running, privileged action requests can be generated.

We’ll use the new OneFS 9.12 Secure Snapshots feature as an example.

Once MPA is enabled on a cluster, any requests to execute a privileged request are paused pending approval.

In this case, an attempt to delete a secure snapshot results in a warning, and the requested action is suspended until approved by an authorizing party.

4. Request Approval.

The privileged action requests can now be approved or rejected, as appropriate.

To private consent for the operation, the authorizing administrator clicks the ‘Approve’ button for the pending request, located under Access > Multi-Party Authorization > Requests:

The authorizing party is prompted for their time-based one-time password (TOTP) security authentication code as part of the approval process:

Or via the CLI:

# isi mpa requests list

# isi mpa requests approve <id> <comment> <approved> --totp-code <******> --approval-valid-before <timestamp> --zone <zone>

5. Finally, the privileged action is ready to be executed.

Once approval has been granted, the WebUI reports a successful approval status.

The privileged secure snapshot delete operation is now able to proceed as expected:

Or from the CLI:

# isi mpa requests view <id>

In the next and final article in this series, we’ll turn our attention to MPA management and troubleshooting.

As we saw in the previous article in this series, OneFS Multi-party Authorization (MPA) requires that one or more additional trusted parties sign off on requested changes for certain privileged actions within a PowerScale cluster.

A prerequisite to enabling MPA in OneFS 9.12 is the requirement for two users with the approval privilege, both registered as internal approvers. Once MPA has been activated on a cluster, any of the predefined ‘privileged actions’ will require explicit approval. In order for a privileged action, such as secure snapshot deletion, to complete, one of the registered MPA approvers must bless the PAA request. Finally, assuming approval was granted, the privileged action can be executed.

The MPA privileged action approval workflow can be summarized in the following request state diagram:

Under the hood, MPA request lifecycle configuration data is stored in the form of key-value pairs, and the MPA Approvers registration status, PAA requests, and request approval details are kept in a shared SQLite database.

Note that MPA approver users can only reside in the System Zone and require the new ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST privileges at a minimum.

MPA includes privileged actions, which extend across the S3 protocol, supporting immutable bucket retention and logging, cluster hardening, and the main one which is the new secure snapshots functionality.

As such, the full complement of these privileged actions in OneFS 9.12 is as follows:

A number of new RBAC privileges are introduced in 9.12 to support MPA, including MPA request, MPA approval, which allows an approver to bless or reject a privileged action request, plus MPA upload, global and access zone settings:

There’s also the new ‘Approval Admin’ default role, and a number of other roles that see the addition of MPA privileges, including SecurityAdmin, SystemAdmin, zone admin, and basic user, etc.

The ApprovalAdmin role is automatically assigned all the required privileges for MPA approval and configuration. For example:

# isi auth roles list | grep -i appr

ApprovalAdmin

# isi auth roles view ApprovalAdmin
       Name: ApprovalAdmin
Description: Allows MPA request approval.
    Members: root
             admin
 Privileges
             ID: ISI_PRIV_LOGIN_PAPI
     Permission: +

             ID: ISI_PRIV_MPA_APPROVAL
     Permission: w

             ID: ISI_PRIV_MPA_REQUEST
     Permission: w

             ID: ISI_PRIV_MPA_SETTINGS_GLOBAL
     Permission: r

             ID: ISI_PRIV_MPA_SETTINGS_ZONE
     Permission: r

The minimum MPA privileges required are ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST.

Here are the system access zone roles that see the addition of MPA privileges in OneFS 9.12:

Similarly, the OneFS 9.12 MPA privileges for non-system zone roles:

MPA approvers can use any of a cluster’s authentication providers, such as file, local, Active Directory or LDAP, and can they grant or reject any MPA request from any zone. However, an approver cannot approve their own request, regardless of whether it’s created by them or created for them.

The CLI commands and platform API endpoints for registration, query, and approval are as follows:

Note that the approver user can easily re-register at any time, say if their TOTP is lost, or for another security concern.

An approval request can be in one of five states: Approved, Cancelled, Completed, Pending, or Rejected.

These states, and their various expiry times, can be viewed with the following CLI command syntax:

# isi mpa settings request-lifecycle list

Approved
    Description: expire in configured days for privilege action execution.
    Expire_time: 7d

Cancelled
    Description: remove from system in configured days.
    Expire_time: 7d

Completed
    Description: remove from system in configured days.
    Expire_time: 30d

Pending
    Description: expire in configured days can be updated by user or approver.
    Expire_time: 7d

Rejected

    Description: remove from system in configured days.
    Expire_time: 30d

The next article in this series will focus on the specific configuration, enablement, and approval procedures for MPA.

Among a handful of new security features, OneFS 9.12 introduces Multi-Party Authorization (MPA). MPA is an administrative approval mechanism that requires at least one additional trusted party to sign off on a requested change, for certain privileged actions within a PowerScale cluster.

As such, MPA helps to mitigate the risk of data loss or system configuration damage from critical actions, by vastly reducing the likelihood of accidental or malicious execution of consequential operations. Conceptually, MPA is fairly analogous to a bank’s safe deposit box, which requires dual control: Both the authorized owner and a bank official must be present in order to open the vault and unlock the box, ensuring that no single party can access its contents independently (dual authorization). MPA enforces a similar security control mechanism wherein operations involving critical or sensitive systems or data require explicit approval from multiple authorized entities. This ensures that no single actor can execute high-impact actions unilaterally, thereby mitigating risk and enhancing operational integrity through enforced oversight.

As such, many enterprises require MPA in order to meet industry data protection requirements, in addition to internal security mandates and best practices.

MPA can be configured and managed from the CLI, WebUI or platform API, but note that the feature is not activated by default. Once MPA has been enabled and configured, all predefined ’privileged actions’ require additional approval from an authorizing user. So this is markedly different from earlier OneFS releases, where the original requester would likely have had the basic rights and been able to execute that action in its entirety themselves.

MPA provides an approval mechanism for a set of ‘privileged actions’, which are defined in OneFS. Additionally, a new default ‘approval admin’ RBAC role is added in OneFS 9.12 specifically for this purpose. This ApprovalAdmin role is automatically assigned all the required privileges for MPA approval and configuration. Specifically:

There are no special licensing requirements for enabling or running MPA. So, upon installation or upgrade to OneFS 9.12, once two or more approval users have been registered, the MPA feature itself can be enabled:

Note that MPA is incompatible with, and therefore not supported on, a PowerScale cluster that is already running in Compliance Mode.

Once MPA is up and running, any approval that’s granted for a privileged operation will remain valid and viable for a defined duration. When approving a request, the approver can specify an expiry time. Once that allotted time has passed, the approved request can no longer be used to execute an action, and a new approval must be generated. If the approver does not specify an expiry time, then, by default, their approval will be valid for seven days.

At a high-level, the MPA workflow is as follows:

1. The first step is registration, which requires two users with the approval privilege, both registered as internal approvers. And note that merely having an RBAC approval privilege is not enough – there is a secure process to register the users. Once there are two registered approval users, the approval admin can enable the MPA feature.

   With MPA is enabled, Privileged Action Approval Requests (PAA) can be generated – either manually, or automatically when a privileged action is attempted. Any of the predefined ‘privileged actions’ will require explicit approval from internal approvers. Next, one of the registered MPA approvers blesses or rejects the PAA Request. And finally, assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

   Step	Action	Description
   1	Approver Registration	Registration requires two users with the approval privilege, both registered as internal approvers. MPA requires a secure process to register the users.
   2	MPA Enablement	Enable MPA via the CLI, WebUI, or platform API. Any of the predefined ‘privileged actions’ will require explicit approval from internal approvers.
   3	Request Creation	Generate a Privileged Action Approval Request (PAA) – either manually, or automatically when a privileged action is attempted.
   4	Request Approval	One of the registered MPA approvers blesses or rejects the PAA request.
   5	Request Execution	Assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

MPA approver users can only reside in the System Zone, and require the new ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST privileges at a minimum. So while a request can be generated from any zone, an approver for that request must reside in the system zone.

An approval request can be in one of the following states:

Note that, in order to approve or reject requests, an approver needs to complete a secure one-time registration.

In OneFS 9.12, the MPA request lifecycle configuration is predefined, and no updates are supported. Only the Approval API (approval_valid_before) can shorten MPA request expiration time in approved status. MPA requests are zone scoped and managed by MPA request lifecycle configuration. An authorized user can view or update a request while it’s in a ‘pending’ state, but only an MPA approver user can move the request to an ‘approved’ or ‘rejected’ state. While an approval is valid (i.e. not expired), the original requester can use it to execute the privileged action. A request stays in an ‘approved’ state until its approval expires, at which point it is moved to a ‘completed’ state.

MPA includes privileged actions, which extend across the S3 protocol, supporting immutable bucket retention and logging, cluster hardening, and the main one which is the new secure snapshots functionality. These are available for selection in the MPA configuration, for example from the WebUI MPA Requests dropdown menu:

In the next article in this series, we’ll look closer at the architectural fundaments that constitute MPA.

Dell PowerScale is already powering up the summer with the launch of the innovative OneFS 9.12 release, which shipped today (14^th August 2025). This new 9.12 release has something for everyone, introducing PowerScale innovations in security, serviceability, reliability, protocols, and ease of use.

OneFS 9.12 represents the latest version of PowerScale’s common software platform for on-premises and cloud deployments. This can make it a excellent choice for traditional file shares and home directories, vertical workloads like M&E, healthcare, life sciences, financial services, plus generative and agentic AI, and other ML/DL and analytics applications.

PowerScale’s scale-out architecture can be deployed on-site, in co-lo facilities, or as customer-managed PowerScale for Amazon AWS and Microsoft Azure deployments, providing core to edge to cloud flexibility, plus the scale and performance needed to run a variety of unstructured workflows on-prem or in the public cloud.

With data security, detection, and monitoring being paramount in this era of unprecedented cyber threats, OneFS 9.12 brings an array of new features and functionality to keep your unstructured data and workloads more available, manageable, and secure than ever.

Protocols

On the S3 object protocol front, OneFS 9.12 sees the debut of new security and immutability functionality. S3 Object Lock extends the standard AWS S3 Object Lock model with PowerScale’s own ‘Bucket-Lock’ protection mode semantics. Object Lock capabilities can operate on a per-zone basis and per-bucket, using the cluster’s compliance clock for the date and time evaluation of object’s retention. Additionally, S3 protocol access logging and bucket logging are also enhanced in this new 9.12 release.

Networking

As part of PowerScale’s seamless protocol failover experience for customers, OneFS 9.12 sees SmartConnect’s default IP allocation method for new pools move to ‘dynamic’. While SMB2 and SMB3 are the primary focus, all protocols benefit from this enhancement, including SMB, NFS, S3, and HDFS. Legacy pools will remain unchanged upon upgrade to 9.12, but any new pools will automatically be provisioned as dynamic (unless manually configured as ‘static’).

Security

In the interests of increased security and ransomware protection, OneFS 9.12 includes new Secure Snapshots functionality. Secure Snapshots provide true snapshot immutability, as well as protection for snapshot schedules, in order to protect against alteration or deletion, either accidentally or by a malicious actor.

Secure snapshots are built upon Multi-party Authorization (MPA), also introduced in OneFS 9.12. MPA prevents an individual administrator from executing privileged operations, such as configuration changes on snapshots and snapshot schedules, by requiring two or more trusted parties to sign off on a requested change for the privileged actions within a PowerScale cluster.

OneFS 9.12 also introduces support for common access cards (CAC) and personal identity verification (PIV) smart cards, providing physical multi-factor authentication (MFA), allowing users to SSH to a PowerScale cluster using the same security badge that grants them access into their office. In addition to US Federal mandates, CAC/PIV integration is a requirement for many security conscious organizations across the public and private sectors.

Upgrade

One-click upgrades in OneFS 9.12 allow a cluster to automatically display and download available  trusted upgrade packages from Dell Support, which can be easily applied via ‘one click installation’ from the OneFS WebUI or CLI. Upgrade package versions are automatically managed by Dell in accordance with a cluster’s telemetry data.

Support

OneFS 9.12 introduces an auto-healing capability, where the cluster detects problems using the HealthCheck framework and automatically executes a repair action for known issues and failures. This helps to increase cluster availability and durability, while reducing the time to resolution and the need for technical support engagements. Furthermore, additional repair-actions can be added at any point, outside of the general OneFS release cycle.

Hardware Innovation

On the platform hardware front, OneFS 9.12 also introduces an HDR Infiniband front-end connectivity option for the PowerScale PA110 performance and backup accelerator. Plus, 9.12 also brings a fast reboot enhancement to the high-memory PowerScale F-series nodes.

In summary, OneFS 9.12 brings the following new features and functionality to the Dell PowerScale ecosystem:

We’ll be taking a deeper look at OneFS 9.12’s new features and functionality in future blog articles over the course of the next few weeks.

Meanwhile, the new OneFS 9.12 code is available on the Dell Support site, as both an upgrade and reimage file, allowing both installation and upgrade of this new release.

For existing clusters running a prior OneFS release, the recommendation is to open a Service Request with to schedule an upgrade. To provide a consistent and positive upgrade experience, Dell is offering assisted upgrades to OneFS 9.12 at no cost to customers with a valid support contract. Please refer to Knowledge Base article KB544296 for additional information on how to initiate the upgrade process.

As we saw in the previous articles in this series, SmartSync in OneFS 9.11 enjoys the addition of backup-to-object functionality, which delivers high performance, full-fidelity incremental replication to ECS, ObjectScale, Wasabi, and AWS S3 & Glacier IR object stores.

This new SmartSync backup-to-object functionality supports the full spectrum of OneFS path lengths, encodings, and file sizes up to 16TB – plus special files and alternate data streams (ADS), symlinks and hardlinks, sparse regions, and POSIX and SMB attributes.

In addition to the standard ‘isi dm’ command set, the following CLI utility can also come in handy for tasks such as verifying the dataset ID for restoration, etc:

# isi_dm browse

For example, to query the SmartSync accounts and datasets:

# isi_dm browse

<no account>:<no dataset> $ list-accounts

000000000000000100000000000000000000000000000000 (tme-tgt)

ec2a72330e825f1b7e68eb2352bfb09fea4f000000000000 (DM Local Account)

fd0000000000000000000000000000000000000000000000 (DM Loopback Account)

<no account>:<no dataset> $ connect-account 000000000000000100000000000000000000000000000000

tme-tgt:<no dataset> $ list-datasets

1       2025-07-22T10:23:33+0000        /ifs/data/zone3

2       2025-07-22T10:23:33+0000        /ifs/data/zone4

1025    2025-07-22T10:25:01+0000        /ifs/data/zone3

2049    2025-07-22T10:30:04+0000        /ifs/data/zone4

tme-tgt:<no dataset> $ connect-dataset 2

tme-tgt:2 </ifs/data/zone4:> $ ls

home                           [dir]

zone2_sync1753179349           [dir]

tme-tgt:2 </ifs/data/zone4:> $ cd zone2_sync1753179349

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ ls

home                           [dir]

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $

Or for additional detail:

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ settings output-to-file-on /tmp/out.txt

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ settings verbose-on

tme-tgt:2 </ifs/data/zone4:zone2_sync1753179349/> $ list-datasets

1       2025-07-22T10:23:33+0000        /ifs/data/zone3 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=1 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=1 } }

2       2025-07-22T10:23:33+0000        /ifs/data/zone4 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=2 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=2 } }

1025    2025-07-22T10:25:01+0000        /ifs/data/zone3 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=1 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=3 } }

2049    2025-07-22T10:30:04+0000        /ifs/data/zone4 { dmdi_tree_id={ dmdti_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdti_local_unid=2 } dmdi_revision={ dmdr_system_guid={dmg_guid=0060486e3954c1b470687f084aa83df6c07d} dmdr_local_unid=4 } }

But when it comes to monitoring and troubleshooting SmartSync, there are a variety of diagnostic tools available. These include:

SmartSync info and errors are typically written to /var/log/isi_dm.log and /var/log/messages, while DM jobs transfer failures generate a log specific to the job ID under /ifs/data/Isilon_Support/datamover/transfer_failures.

Once a policy is running, the job status is reported via ‘isi dm jobs list’. Once complete, job histories are available by running ‘isi dm historical jobs list’. More details for a specific job can be gleaned from the ‘isi dm job view’ command, using the pertinent job ID from the list output above. Additionally, the ‘isi_datamover_job_status’ command with the job ID as an argument will also supply detailed information about a specific job.

Once running, a DM job can be further controlled via the ‘isi dm jobs modify’ command, and available actions include cancel, partial-completion, pause, or resume.

If a certificate authority (CA) is not correctly configured on a PowerScale cluster, the SmartSync daemon will not start, even though accounts and policies can still be configured. Be aware that the failed policies will not be reported via ‘isi dm jobs list’ or ‘isi dm historical-jobs list’ since they never started. Instead, an improperly configured CA is reported in the /var/log/isi_dm.log as follows:

Certificates not correctly installed, Data Mover service sleeping: At least one CA must be installed: No such file or directory from dm_load_certs_from_store (/b/mnt/src/isilon/lib/isi_dm/isi_dm_remote/src/rpc/dm_tls.cpp:197 ) from dm_tls_init (/b/mnt/src/isilon/lib/isi_dm/isi_dm_remote/src/rpc/dm_tls.cpp:279 ): Unable to load certificate information

Once a CA and identity are correctly configured, the SmartSync service automatically activates. Next, SmartSync attempts a handshake with the target. If the CA or identity is mis-configured, the handshake process fails, and generates an entry in /var/log/isi_dm.log. For example:

2025-07-30T12:38:17.864181+00:00 GEN-HOP-NOCL-RR-1(id1) isi_dm_d[52758]: [0x828c0a110]: /b/mnt/src/isilon/lib/isi_dm/isi_dm_remote/src/acct_mon.cpp:dm_acc tmon_try_ping:348: [Fiber 3778] ping for account guid: 0000000000000000c4000000000000000000000000000000, result: dead

Note that the full handshake error detail is logged if the SmartSync service (isi_dm_d) is set to log at the ‘info’ or ‘debug’ level using isi_ilog:

# isi_ilog -a isi_dm_d --level info+

Valid ilog levels include:

fatal error err notice info debug trace

error+ err+ notice+ info+ debug+ trace+

A copy or repeat-copy policy requires an available dataset for replication before running. If a dataset has not been successfully created prior to the copy or repeat-copy policy job starting for the same base path, the job is paused. In the following example, the base path of the copy policy is not the same as that of the dataset policy, hence the job fails with a “path doesn’t match…” error.

# ls -l /ifs/data/Isilon_support/Datamover/transfer_failures

Total 9

-rw-rw----   1 root  wheel  679  July 20 10:56 baseline_failure_10

# cat /ifs/data/Isilon_support/Datamover/transfer_failures/baseline_failure_10

Task_id=0x00000000000000ce, task_type=root task ds base copy, task_state=failed-fatal path doesn’t match dataset base path: ‘/ifs/test’ != /ifs/data/repeat-copy’:

from bc_task)initialize_dsh (/b/mnt/src/isilon/lib/isi_dm/isi_dm/src/ds_base_copy

from dmt_execute (/b/mnt/src/isilon/lib/isi_dm/isi_dm/src/ds_base_copy_root_task

from dm_txn_execute_internal (/b/mnt/src/isilon/lib/isi_dm/isi_dm_base/src/txn.cp

from dm_txn_execute (/b/mnt/src/isilon/lib/isi_dm/isi_dm_base/src/txn.cpp:2274)

from dmp_task_spark_execute (/b/mnt/src/isilon/lib/isi_dm/isi_dm/src/task_runner.

Once any errors for a policy have been resolved, the ‘isi dm jobs modify’ command can be used to resume the job.

As we saw in the previous article in this series, SmartSync in OneFS 9.11 sees the addition of backup-to-object, which provides high performance, full-fidelity incremental replication to ECS, ObjectScale, Wasabi, and AWS S3 & Glacier IR object stores.

This new SmartSync backup-to-object functionality supports the full spectrum of OneFS path lengths, encodings, and file sizes up to 16TB – plus special files and alternate data streams (ADS), symlinks and hardlinks, sparse regions, and POSIX and SMB attributes. Specifically:

SmartSync backup-to-object operates on user-defined data set, which are essentially OneFS file system snapshots with plus additional properties.

A data set creation policy takes snapshots and creates a data set out of it. Additionally, there are also copy and repeat copy policies which are the policies that will transfer that data set to another system. And the execution of these two policy types can be linked and scheduled separately. So you can have one schedule for your data set creation, say to create a data set every hour on a particular path. And you can have a tiered or different distribution system for the actual copy itself. For example, to copy every hour to a hot DR cluster in data center A. But also copy every month to a deep archive cluster in data center B. So all these things are possible now, without increasing the bloat of snapshots on the system, since they’re now able to be shared.

Currently, SmartSync does not have a WebUI presence, so all its configuration is either via the command-line or platform API.

Here’s the procedure for crafting a baseline replication config:

Essentially, create the replication account, which in OneFS 9.11 will be either Dell ECS or Amazon AWS. Then configure that dataset creation policy, run it, and, if desired, create a repeat-copy policy. These specific steps with their CLI syntax include:

1. Create a replication account:

# isi dm account create --account-type [AWS_S3 | ECS_S3]

2. Configure a dataset creation policy

# isi dm policies create [Policy Name] --policy-type CREATION

3. Run the dataset creation policy:

# isi dm policies list

# isi dm policies modify [Creation policy id] –-run-now=true

# isi dm jobs list

# isi dm datasets list

4. create a repeat-copy policy

# isi dm policies create [Policy Name] --policy-type=' REPEAT_COPY'

5. Run the repeat-copy policy:

# isi dm policies list

# isi dm policies modify [Repeat-copy policy id] –-run-now=true

6. View the data replication job status

# isi dm jobs list

Similarly for an incremental replication config:

Note that the dataset creation policy and repeat-copy policy are already created in the baseline replication configure and can be ignored.

Incremental replication using the dataset create and repeat-copy policies from the previous slide’s baseline config.

1. Run the dataset creation policy

# isi dm policies list

# isi dm policies modify [Creation policy id] –-run-now=true

# isi dm jobs list

# isi dm datasets list

2. Run the repeat-copy policy:

# isi dm policies list

# isi dm policies modify [Repeat-copy policy id] –-run-now=true

3. View the data replication incremental job status

# isi dm jobs list

And here’s the basic procedure for creating and running a partial or full restore:

Note that the replication account is already created on the original cluster and the creation step can be ignored.  Replication account creation is only required if restoring the dataset to a new cluster.

Additionally, partial restoration involves a subset of the directory structure, specified via the ‘source path’ , whereas full restoration invokes a restore of the entire dataset.

The process includes creating the replication account if needed, finding the ID of the dataset to be restored, creating and running the partial or full restoration policy, and checking the job status to verify it ran successfully.

1. Create a replication account:

# isi dm account create --account-type [AWS_S3 | ECS_S3]

For example:

# isi dm account create --account-type ECS_S3 --name [Account Name] --access-id [access-id] --uri [URI with bucket-name] --auth-mode CLOUD --secret-key [secret-key] --storage-class=[For AWS_S3 only: STANDARD or GLACIER_IR]

2. Verify the dataset ID for restoration:

# isi_dm browse

Checking the following attributes:

• list-accounts
• connect-account [Source Account ID created in step 1]
• list-datasets
• connect-dataset [Dataset id]

3. Create a partial or full restoration policy

# isi dm policies create [Policy Name] --policy-type='COPY'

4. Run the partial or full restoration policy:

# isi dm policies modify [Restoration policy id] –-run-now=true

5. View the data restoration job status

# isi dm jobs list

OneFS 9.11 also introduces recovery point objective or RPO alerts for SmartSync, but note that these are for repeat-copy policies only. These RPO alerts can be configured through the replication policy by adding the desired time value to the ‘repeat-copy-rpo-alert’ parameter. If this configured threshold is exceeded, an RPO alert is triggered. This RPO alert is automatically resolved after the next successful policy job run.

Also be aware that the default time value for a repeat copy RPO is zero, which instructs SmartSync to not generate RPO alerts for that policy.

The following CLI syntax can be used to create a replication policy, with the ‘–repeat-copy-rpo-alert’ flag set for the desired time:

# isi dm policies create [Policy Name] --policy-type=' REPEAT_COPY' --enabled='true' --priority='NORMAL' --repeat-copy-source-base-path=[Source Path] --repeat-copy-base-base-account-id=[Source account id] --repeat-copy-base-source-account-id=[Source account id] --repeat-copy-base-target-account-id=[Target account id] --repeat-copy-base-new-tasks-account=[Source account id] --repeat-copy-base-target-dataset-type='FILE_ON_OBJECT_BACKUP' --repeat-copy-base-target-base-path=[Bucket Name] --repeat-copy-rpo-alert=[time]

And similarly to change the RPO alert configuration on an existing replication policy:

# isi dm policies modify [Policy id] --repeat-copy-rpo-alert=[time]

An alert is triggered and corresponding CELOG event created if the specified RPO for the policy is exceeded. For example:

# isi event list

ID   Started     Ended       Causes Short                     Lnn  Events  Severity

--------------------------------------------------------------------------------------

1898 07/15 00:00 07/15 00:00 SW_CELOG_HEARTBEAT               1    1       information

2012 07/15 06:03 --          SW_DM_RPO_EXCEEDED               2    1       warning

--------------------------------------------------------------------------------------

And then once RPO alert has been resolved after a successful replication policy job run:

# isi event list

ID   Started     Ended       Causes Short                     Lnn  Events  Severity

--------------------------------------------------------------------------------------

1898 07/15 00:00 07/15 00:00 SW_CELOG_HEARTBEAT               1    1       information

2012 07/15 06:03 07/15 06:12 SW_DM_RPO_EXCEEDED               2    2       warning

--------------------------------------------------------------------------------------

Another significant benefactor of new functionality in the recent OneFS 9.11 release is SmartSync. As you may recall, SmartSync allows multiple copies of a dataset to be copied, replicated, and stored across locations and regions, both on and off-prem, providing increased data resilience and the ability to rapidly recover from catastrophic events.

In addition to fast, efficient, scalable protection with granular recovery, SmartSync allows organizations to utilize lower cost object storage as the target for backups, reduce data protection complexity and cost by eliminating the need for separate backup applications. Plus disaster recovery options include restoring a dataset to its original state, or cloning a new cluster.

SmartSync sees the following enhancements in OneFS 9.11:

• Automated incremental-forever replication to object storage.
• Unparalleled scalability and speed, with seamless pause/resume for robust resiliency and control.
• End-to-end encryption for security of data-in-flight and at rest.
• Complete data replication, including soft/hard links, full file paths, and sparse files
• Object storage targets: AWS S3, AWS Glacier IR, Dell ECS/ObjectScale, and Wasabi (with the addition of Azure and GCP support in a future release).

But first, a bit of background. Introduced back in OneFS 9.4, SmartSync operates in two distinct modes:

• Regular push-and-pull transfer of file data between PowerScale clusters.
• CloudCopy, copying of file-to-object data from a source cluster to a cloud object storage target.

CloudCopy copy-to-object in OneFS 9.10 and earlier releases is strictly a one-time copy tool, rather than a replication utility. So, after a copy, viewing the bucket contents from AWS, console or S3 browser yielded an object format tree-like representation of the OneFS file system. However, there were a number of significant shortcomings, such as no native support for attributes like ACLs, or certain file types like character files, and no method to represent hard links in a reasonable way. So OneFS had to work around these things by expanding hard links, and redirecting objects that had too long of a path. The other major limitation was that it really had just been a one-and-done copy. After creating and running a policy, once the job had completed the data was in the cloud, and that was it. OneFS had no provision for any incremental transfer of any subsequent changes to the cloud copy when the source data changed.

In order to address these limitations, SmartSync in OneFS 9.11 sees the addition of backup-to-object functionality. This includes a full-fidelity file system baseline, plus fast incremental replication to Dell ECS and ObjectScale, Wasabi, and AWS S3 and Glacier IR object stores.

This new backup-to-object functionality supports the full range of OneFS path lengths, encodings, and file sizes up to 16TB – plus special files and alternate data streams (ADS), symlinks and hardlinks, sparse regions, and POSIX and SMB attributes.

Architecturally, SmartSync incorporates the following concepts:

Under the hood, SmartSync uses this concept of a data set, which is fundamentally an abstraction of a OneFS file system snapshot – albeit with some additional properties attached to it.

Each data set is identified by a unique ID. Plus, with this notion of data sets, OneFS can now also perform an A to B replication and an A to C replication – two replications of the same data set to two different targets. Plus with these new data sets, B and C can now also reference each other and perform incremental replication amongst themselves, assuming they have a common ancestor snapshot that they share.

A SmartSync data set creation policy takes snapshots and creates a data set from it. Additionally, there are also copy and repeat copy policies, which are the policies that are used to transfer that data set to another system. The execution of these two policy types can be linked and scheduled separately. So one schedule can be for data set creation, say to create a data set every hour on a particular path, and the other schedule for a tiered or different distribution system for the actual copy itself. For example, in order to copy hourly to a hot DR cluster in data center A, and also copy monthly to a deep archive cluster in data center B – all without increasing the proliferation of snapshots on the system, since they’re now able to be shared.

Additionally, SmartSync in 9.11 also introduces the foundational concept of a global file ID (GFID), which is essentially a global LIN that represents a specific file on a particular system. OneFS can now use this GFID, in combination with a data set, to reference a file anywhere and guarantee that it means the same thing across every cluster.

Security-wise, each SmartSync daemon has an identity certificate that acts as both a client and server certificate depending on the direction of the data movement. This identity certificate is signed by a non-public certificate authority. To establish trust between two clusters, they must have each other’s CAs. These CAs may be the same. Trust groups (daemons that may establish connections to each other) are formed by having shared CAs installed.

There are no usernames or passwords; authentication is authorization for V1. All cluster-to-cluster communication is performed via TLS-encrypted traffic. If absolutely necessary, encryption (but not authorization) can be disabled by setting a ‘NULL’ encryption cipher for specific use cases that require unencrypted traffic.

The SmartSync daemon supports checking certificate revocation status via the Online Certificate Status Protocol (OCSP). If the cluster is hardened and/or in FIPS-compliant mode, OCSP checking is forcibly enabled and set to the Strict stringency level, where any failure in OCSP processing results in a failed TLS handshake. Otherwise, OCSP checking can be totally disabled or set to a variety of values corresponding to desired behavior in cases where the responder is unavailable, the responder does not have information about the cert in question, and where information about the responder is missing entirely. Similarly, an override OCSP responder URI is configurable to support cases where preexisting certificates do not contain responder information.

SmartSync also supports a ‘strict hostname check’ option which mandates that the common name and/or subject alternative name fields of the peer certificate match the URI used to connect to that peer. This option, along with strict OCSP checking and disabling the null cipher option, are forcibly set when the cluster is operating in a hardened or FIPS-compliant mode.

For object storage connections, SmartSync uses ‘isi_cloud_api’ just as CloudPools does. As such, all considerations that apply to CloudPools also apply to SmartSync as well.

In the next article in this series, we’ll turn our attention to the core architecture and configuration of SmartSync backup-to-object.

The last couple of articles generated several questions for the field around durability and resilience in the newly released PowerScale H710/0 and A310/0 nodes. In this article, we’ll take a deeper look at the OneFS journal and boot drive mirroring functionality in these H and A-series platforms.

PowerScale chassis-based hardware, such as the new H710/7100 and A310/3100, stores the local filesystem journal and its mirror on persistent, battery-backed flash media within each node, with a 4RU PowerScale chassis housing four nodes. These nodes comprise a ‘compute node enclosure for the CPU, memory, and, and network cards, plus associated drive containers, or sleds, for each node.

The PowerScale H and A-series employ a node-pair architecture to dramatically increased system reliability, with each pair of nodes residing within a chassis power zone. This means that if a node’s PSU fails, the peer PSU supplies redundant power. It also drives a minimum cluster or node pool size of four nodes (one chassis) for the PowerScale H and A-series platforms, pairwise node population, and the need to scale the cluster two nodes at a time.

A node’s file system journal is protected against sudden power loss or hardware failure by OneFS’ journal vault functionality – otherwise known as ‘powerfail memory persistence’, or PMP. PMP automatically stores both the local journal and journal mirror on a separate flash drive across both nodes in a node pair:

This journal de-staging process is known as ‘vaulting’, during which the journal is protected by a dedicated battery in each node until it’s safely written from DRAM to SSD on both nodes in a node-pair. With PMP, constant power isn’t required to protect the journal in a degraded state since the journal is saved to M.2 flash and mirrored on the partner node.

So, the mirrored journal is comprised of both hardware and software components, including the following constituent parts:

Journal Hardware Components

• System DRAM
• 2 Vault Flash
• Battery Backup Unit (BBU)
• Non-Transparent Bridge (NTB) PCIe link to partner node
• Clean copy on disk

Journal Software Components

• Power-fail Memory Persistence (PMP)
• Mirrored Non-volatile Interface (MNVI)
• IFS Journal + Node State Block (NSB)
• Utilities

Asynchronous DRAM Refresh (ADR) preserves RAM contents when the operating system is not running. ADR is important for preserving RAM journal contents across reboots, and it does not require any software coordination to do so.

The journal vaulting functionality encompasses the hardware, firmware, and operating system, ensuring that the journal’s contents are preserved across power failure. The mechanism is similar to the software journal mirroring employed on the PowerScale F-series nodes, albeit using a PCIe-based NTB on the chassis based platforms, instead of using the back-end network as with the all-flash nodes.

On power failure, the PMP vaulting functionality is responsible for copying both the local journal and the local copy of the partner node’s journal to persistent flash. On restoration of power, PMP is responsible for restoring the contents of both journals from flash to RAM, and notifying the operating system.

A single dedicated 480GB NVMe flash device (nvd0) is attached via an M.2 slot on the motherboard of the H710/0 and A310/0 node’s compute module, residing under the battery backup unit (BBU) pack.

This is in contrast to the prior H and A-series chassis generations, which used a 128GB SATA M.2 device (/dev/ada0).

For example, the following CLI commands show the NVMe M.2 flash device in an A310 node:

# isi_hw_status | grep -i prod
Product: A310-4U-Single-96GB-1x1GE-2x25GE SFP+-60TB-1638GB SSD-SED

# nvmecontrol devlist
 nvme0: Dell DN NVMe FIPS 7400 RI M.2 80 480GB
    nvme0ns1 (447GB)

# gpart show | grep nvd0
=>       40  937703008  nvd0  GPT  (447G)

# gpart show -l nvd0
=>       40  937703008  nvd0  GPT  (447G)
         40       2008        - free -  (1.0M)
       2048   41943040     1  isilon-pmp  (20G)
   41945088  895757960        - free -  (427G)

In the above, the ‘isilon-pmp’ partition on the M.2 flash device is used by the file system journal for its vaulting activities.

The the NVMe M.2 device is housed on the node compute module’s riser card, and its firmware is managed by the OneFS DSP (drive support package) framework:

Note that the entire compute module must be removed in order for its M.2 flash to be serviced. If the M.2 flash does need to be replaced for any reason, it will be properly partitioned and the PMP structure will be created as part of arming the node for vaulting.

For clusters using data-at-rest encryption (DARE), an encrypted M.2 device is used, in conjunction with SED data drives, to provide full FIPS compliance.

The battery backup unit (BBU), when fully charged, provides enough power to vault both the local and partner journal during a power failure event:

A single battery is utilized in the BBU, which also supports back-to-back vaulting:

On the software side, the journal’s Power-fail Memory Persistence (PMP) provides an equivalent to the NVRAM controller‘s vault/restore capabilities to preserve Journal. The PMP partition on the M.2 flash drive provides an interface between the OS and firmware.

If a node boots and its primary journal is found to be invalid for whatever reason, it has three paths for recourse:

• Recover journal from its M.2 vault.
• Recover journal from its disk backup copy.
• Recover journal from its partner node’s mirrored copy.

The mirrored journal must guard against rolling back to a stale copy of the journal on reboot. This necessitates storing information about the state of journal copies outside the journal. As such, the Node State Block (NSB) is a persistent disk block that stores local and remote journal status (clean/dirty, valid/invalid, etc), as well as other non-journal information. NSB stores this node status outside the journal itself, and ensures that a node does not revert to a stale copy of the journal upon reboot.

Here’s the detail of an individual node’s compute module:

Of particular note is the ‘journal active’ LED, which is displayed as a white ‘hand icon’:

When this white hand icon is illuminated, it indicates that the mirrored journal is actively vaulting, and it is not safe to remove the node!

There is also a blue ‘power’ LED, and a yellow ‘fault’ LED per node. If the blue LED is off, the node may still be in standby mode, in which case it may still be possible to pull debug information from the baseboard management controller (BMC).

The flashing yellow ‘fault’ LED has several state indication frequencies:

The mirrored non-volatile interface (MNVI) sits below /ifs and above RAM and the NTB, providing the abstraction of a reliable memory device to the /ifs journal. MNVI is responsible for synchronizing journal contents to peer node RAM, at the direction of the journal, and persisting writes to both systems while in a paired state. It upcalls into the journal on NTB link events, and notifies the journal of operation completion (mirror sync, block IO, etc). For example, when rebooting after a power outage, a node automatically loads the MNVI. It then establishes a link with its partner node and synchronizes its journal mirror across the PCIe Non-Transparent Bridge (NTB).

The Non-transparent Bridge (NTB) connects node pairs for OneFS Journal Replica:

The NTB Link itself is PCIe Gen3 X8, but there is no guarantee of NTB interoperability between different CPU generations. As such, the H710/0 and A310/0 use version 4 of the NTB driver, whereas the previous hardware generation uses NTBv3. This therefore means mixed-generation node pairs are unsupported.

Prior to mounting the /ifs file system, OneFS locates a valid copy of the journal from one of the following locations in order of preference:

Assuming the node was shut down cleanly, it will boot using a local disk copy of the journal. The journal will be restored into DRAM and /ifs will mount. On the other hand, if the node suffered a power disruption, the journal will be restored into DRAM from the M.2 vault flash instead (the PMP copies the journal into the M.2 vault during a power failure).

In the event that OneFS is unable to locate a valid journal on either the hard drives or M.2 flash on a node, it will retrieve a mirrored copy of the journal from its partner node over the NTB.  This is referred to as ‘Sync-back’.

Note: Sync-back state only occurs when attempting to mount /ifs.

On booting, if a node detects that its journal mirror on the partner node is out of sync (invalid), but the local journal is clean, /ifs will continue to mount.  Subsequent writes are then copied to the remote journal in a process known as ‘sync-forward’.

Here’s a list of the primary journal states:

During normal operation, writes to the primary journal and its mirror are managed by the MNVI device module, which writes through local memory to the partner node’s journal via the NTB. If the NTB is unavailable for an extended period, write operations can still be completed successfully on each node. For example, if the NTB link goes down in the middle of a write operation, the local journal write operation will complete. Read operations are processed from local memory.

Additional journal protection for PowerScale chassis-based platforms is provided by OneFS’ powerfail memory persistence (PMP) functionality, which guards against PCI bus errors that can cause the NTB to fail.  If an error is detected, the CPU requests a ‘persistent reset’, during which the memory state is protected and node rebooted. When back up again, the journal is marked as intact and no further repair action is needed.

If a node loses power, the hardware notifies the BMC, initiating a memory persistent shutdown.  At this point the node is running on battery power. The node is forced to reboot and load the PMP module, which preserves its local journal and its partner’s mirrored journal by storing them on M.2 flash.  The PMP module then disables the battery and powers itself off.

Once power is back on and the node restarted, the PMP module first restores the journal before attempting to mount /ifs.  Once done, the node then continues through system boot, validating the journal, setting sync-forward or sync-back states, etc.

The mirrored journal has the following CLI commands, although these should seldom be needed during normal cluster operation:

• isi_save_journal
• isi_checkjournal
• isi_testjournal
• isi_pmp

A node’s journal can be checked and confirmed healthy as follows:

# isi_testjournal
Checking One external batteries Health...
Batteries good
Checking PowerScale Journal integrity...
Mounted DRAM journal check: good
IFS is mounted.

During boot, isi_checkjournal and isi_testjournal will invoke isi_pmp. If the M.2 vault devices are unformatted, isi_pmp will format the devices.

On clean shutdown, isi_save_journal stashes a backup copy of the /dev/mnv0 device on the root filesystem, just as it does for the NVRAM journals in previous generations of hardware.

If a mirrored journal issue is suspected, or notified via cluster alerts, the best place to start troubleshooting is to take a look at the node’s log events. The journal logs to /var/log/messages, with entries tagged as ‘journal_mirror’.

Additionally, the following sysctls also provide information about the state of the journal mirror itself and the MNVI connection respectively:

# sysctl efs.journal.mirror_state
efs.journal.mirror_state:
{
    Journal state: valid_protected
    Journal Read-only: false
    Need to inval mirror: false
    Sync in progress: false
    Sync error: 0
    Sync noop in progress: false
    Mirror work queued: false
    Local state:
    {
        Clean: dirty
        Valid: valid
    }
    Mirror state:
    {
        Connection: up
        Validity: valid
    }
}

And the MNVI connection state:

# sysctl hw.mnv0.state
hw.mnv0.state.iocnt: 0
hw.mnv0.state.cb_active: 0
hw.mnv0.state.io_gate: 0
hw.mnv0.state.state: 3

OneFS provides the following CELOG events for monitoring and alerting about mirrored journal issues:

Another OneFS reliability optimization for the PowerScale chassis-based platforms is boot partition mirroring. OneFS boot and other OS partitions are stored on a node’s internal drives, and these partitions are mirrored (with the exception of crash dump partitions). The two mirrors protect against disk sled removal. Since each drive in a disk sled belongs to a separate disk pool, both elements of a mirror cannot live on the same sled.

With regard to the nodes’ internal drives, the boot disk reservation size has increased to 18GB on these new platforms from 8GB on the previous generation. Plus partition sizes have also been expanded on these new platforms in OneFS 9.11, as follows:

OneFS automatically rebalances these mirrors in anticipation of, and in response to, service events. Mirror rebalancing is triggered by drive events such as suspend, softfail and hard loss.

The ‘isi_mirrorctl verify’ and ‘gmirror status’ CLI commands can be used to confirm that boot mirroring is working as intended. For example, on an A310 node:

# gmirror status
Name Status Components
mirror/root0 COMPLETE da10p3 (ACTIVE)
da11p3 (ACTIVE)
mirror/mfg COMPLETE da15p7 (ACTIVE)
da12p6 (ACTIVE)
mirror/kernelsdump COMPLETE da15p6 (ACTIVE)
mirror/kerneldump COMPLETE da15p5 (ACTIVE)
mirror/var-crash COMPLETE da15p3 (ACTIVE)
da9p3 (ACTIVE)
mirror/journal-backup COMPLETE da14p5 (ACTIVE)
da12p5 (ACTIVE)
mirror/jbackup-peer COMPLETE da14p3 (ACTIVE)
da12p3 (ACTIVE)
mirror/keystore COMPLETE da12p7 (ACTIVE)
da10p10 (ACTIVE)
mirror/root1 COMPLETE da11p7 (ACTIVE)
da10p7 (ACTIVE)
mirror/var0 COMPLETE da11p6 (ACTIVE)
da10p6 (ACTIVE)
mirror/hw COMPLETE da10p9 (ACTIVE)
da7p5 (ACTIVE)
mirror/var1 COMPLETE da10p8 (ACTIVE)
da7p3 (ACTIVE)

Or:

# isi_mirrorctl verify
isi.sys.distmirror - INFO - Mirror root1: has an ACTIVE consumer of da11p5
isi.sys.distmirror - INFO - Mirror root1: has an ACTIVE consumer of da10p7
isi.sys.distmirror - INFO - Mirror var1: has an ACTIVE consumer of da13p5
isi.sys.distmirror - INFO - Mirror var1: has an ACTIVE consumer of da16p5
isi.sys.distmirror - INFO - Mirror journal-backup: has an ACTIVE consumer of da12p5
isi.sys.distmirror - INFO - Mirror journal-backup: has an ACTIVE consumer of da16p6
isi.sys.distmirror - INFO - Mirror jbackup-peer: has an ACTIVE consumer of da12p3
isi.sys.distmirror - INFO - Mirror jbackup-peer: has an ACTIVE consumer of da14p3
isi.sys.distmirror - INFO - Mirror var-crash: has an ACTIVE consumer of da10p6
isi.sys.distmirror - INFO - Mirror var-crash: has an ACTIVE consumer of da11p3
isi.sys.distmirror - INFO - Mirror kerneldump: has an ACTIVE consumer of da14p5
isi.sys.distmirror - INFO - Mirror root0: has an ACTIVE consumer of da10p3
isi.sys.distmirror - INFO - Mirror root0: has an ACTIVE consumer of da13p6
isi.sys.distmirror - INFO - Mirror var0: has an ACTIVE consumer of da13p3
isi.sys.distmirror - INFO - Mirror var0: has an ACTIVE consumer of da16p3
isi.sys.distmirror - INFO - Mirror kernelsdump: has an ACTIVE consumer of da14p6
isi.sys.distmirror - INFO - Mirror mfg: has an ACTIVE consumer of da13p9
isi.sys.distmirror - INFO - Mirror mfg: has an ACTIVE consumer of da16p7
isi.sys.distmirror - INFO - Mirror hw: has an ACTIVE consumer of da10p8
isi.sys.distmirror - INFO - Mirror hw: has an ACTIVE consumer of da13p8
isi.sys.distmirror - INFO - Mirror keystore: has an ACTIVE consumer of da13p10
isi.sys.distmirror - INFO - Mirror keystore: has an ACTIVE consumer of da16p8

The A310 node’s disks in the output above are laid out as follows:

# isi devices drive list
Lnn  Location  Device    Lnum  State   Serial       Sled
---------------------------------------------------------
128  Bay  1    /dev/da1  15    L3      X3X0A0JFTMSJ N/A
128  Bay  2    -         N/A   EMPTY                N/A
128  Bay  A0   /dev/da4  12    HEALTHY WQB0QKBR     A
128  Bay  A1   /dev/da3  13    HEALTHY WQB0QHV4     A
128  Bay  A2   /dev/da2  14    HEALTHY WQB0QHN3     A
128  Bay  B0   /dev/da7  9     HEALTHY WQB0QH4S     B
128  Bay  B1   /dev/da6  10    HEALTHY WQB0QGY3     B
128  Bay  B2   /dev/da5  11    HEALTHY WQB0QJWE     B
128  Bay  C0   /dev/da10 6     HEALTHY WQB0QJ26     C
128  Bay  C1   /dev/da9  7     HEALTHY WQB0QHYW     C
128  Bay  C2   /dev/da8  8     HEALTHY WQB0QK6Q     C
128  Bay  D0   /dev/da13 3     HEALTHY WQB0QJES     D
128  Bay  D1   /dev/da12 4     HEALTHY WQB0QHGG     D
128  Bay  D2   /dev/da11 5     HEALTHY WQB0QKH5     D
128  Bay  E0   /dev/da16 0     HEALTHY WQB0QHFR     E
128  Bay  E1   /dev/da15 1     HEALTHY WQB0QJWD     E
128  Bay  E2   /dev/da14 2     HEALTHY WQB0QKGB     E
---------------------------------------------------------

When it comes to SmartFailing nodes, there are a couple of additional caveats to be aware of with mirrored journal and the PowerScale chassis-based platforms:

• When SmartFailing one node in a pair, there is no compulsion to smartfail its partner node too.
• A node will still run indefinitely with its partner absent. However, this significantly increases the window of risk since there is no journal mirror to rely on (in addition to lack of redundant power supply, etc).
• If a single node in a pair is SmartFailed, the other node’s journal is still protected by the vault and powerfail memory persistence.