Tag: Cluster

OneFS Root Lockdown Mode

Tucked among the plethora of capabilities that debut in OneFS 9.12 is Root Lockdown Mode (RLM), a security feature designed to disable and block access to the cluster’s root account. This new RLM functionality enforces role-based access controls (RBAC) for cluster administrators, thereby reducing the risk of a single privileged user bypassing security policies and performing potentially destructive operations on the cluster configuration or its data. Given that the root account has inherent, unaudited permissions to override security controls, restricting its access can significantly strengthen PowerScale’s security posture.

RLM is managed through the OneFS hardening engine, which is part of a cluster’s compliance mode framework.

RLM leverages the OneFS compliance mode protections provided by the hardening engine and requires both the ‘Hardening’ and ‘SmartLock’ licenses to function. It introduces a new hardening profile, aptly named ‘root-lockdown’, which can be applied or disabled as desired. However, transitioning between RLM states does involve a complete cluster reboot. Plus, if MPA is active, the operation must be approved by multiple parties. OneFS services, and admins, can inspect ‘sysctl’ values to determine whether the cluster is operating in Compliance Mode or RLM.

Note that, unlike MPA, RLM in OneFS 9.12, does not have a WebUI interface, and so is only configurable via the command line (CLI).

Under the hood, when hardening is activated, the security hardening engine reads the desired configuration, such as RLM, from its config files. Sets of rules, or config items, are applied to the hardening configuration to increase security and, in the case of RML, ensure disablement of a cluster’s root account. These rules are grouped by profile, which contain collections of named rules. Profiles are stored in separate .xml files under /etc/isi_hardening/profiles.

# ls /etc/isi_hardening/profiles
profile_root-lockdown.xml       profile_stig.xml

As of OneFS 9.12, there are two hardening profiles available:

  • Root Lockdown Mode (RLM)
  • STIG compliance

Plus, the hardening infrastructure is in place to support additional profiles as and when they are required.

Individual rules are stored in separate .xml files under /etc/isi_hardening/rules:

# ls /etc/isi_hardening/rules

rules_apache.xml rules_celog.xml rules_root.xml

rules_audit.xml rules_fips.xml rules_shell_timeout.xml

rules_auth.xml rules_misc.xml rules_umask.xml

rules_banners.xml rules_password.xml

rules_cacpiv.xml rules_pki_ocsp.xml

For example, the ‘rules_root.xml’ file for disabling the root account:

# cat rules_root.xml
<!-- Copyright (c) 2024-2025 Dell Inc. or its subsidiaries. All Rights Reserved. -->
<CONFIG_ITEMS>
  <CONFIG_ITEM id="disable_root_account" version="1">
    <PapiOperation>
      <DO>
        <URI>/auth/users/root</URI>
        <BODY>{"enabled": false}</BODY>
        <KEY>users</KEY>
      </DO>
      <UNDO>
        <URI>/auth/users/root</URI>
        <BODY>{"enabled": true}</BODY>
        <KEY>users</KEY>
      </UNDO>
      <ACTION_SCOPE>CLUSTER</ACTION_SCOPE>
      <IGNORE>FALSE</IGNORE>
    </PapiOperation>
  </CONFIG_ITEM>
  <CONFIG_ITEM id="root_shell_nologin" version="1">
    <PapiOperation>
      <DO>
        <URI>/auth/users/root</URI>
        <BODY>{"shell": "/sbin/nologin"}</BODY>
        <KEY>users</KEY>
      </DO>
      <UNDO>
        <URI>/auth/users/root</URI>
        <BODY>{"shell": "/usr/local/bin/zsh"}</BODY>
        <KEY>users</KEY>
      </UNDO>
      <ACTION_SCOPE>CLUSTER</ACTION_SCOPE>
      <IGNORE>FALSE</IGNORE>
    </PapiOperation>
  </CONFIG_ITEM>
</CONFIG_ITEMS>

Rules are grouped by functional area affected, and can apply to platform API configuration ‘collections’. For example, a rule can be applied to all NFS exports or all SyncIQ policies. In addition to actionable rules, ‘check-only’ rules are supported which apply no changes.

The hardening engine rules allow comparator logic in addition to equality, and can evaluate conditions like whether a string is empty or non-empty, and if a given timeout is greater or equal to the required value. The RLM profile specifies the following hardening rule set:

# cat /etc/isi_hardening/profiles/profile_root-lockdown.xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- Copyright (c) 2024-2025 Dell Inc. or its subsidiaries. All Rights Reserved. -->
<Profiles version="1">
    <Profile>
        <Name>root-lockdown</Name>
        <Description>Enable security settings that disable the root account.</Description>
        <Rule>set_rlm_cluster_wide</Rule>
        <Rule>require_disabled_STIG_profile</Rule>
        <Rule>require_password_single_user_mode</Rule>
        <Rule>require_rlm_aclr_users</Rule>
        <Rule>require_rlm_cluster_state</Rule>
        <Rule>disable_kdb</Rule>
        <Rule>disable_root_account</Rule>
        <Rule>root_shell_nologin</Rule>
        <Rule>check_disable_root_account</Rule>
        <Rule>check_root_shell_nologin</Rule>
        <Rule>check_rlm_sysctls</Rule>
    </Profile>
</Profiles>

Most of the above RLM rule definitions live in the ‘/etc/isi_hardening/rules/rules_misc.xml’ file. For example:

# grep -A 10 "disable_root" /etc/isi_hardening/rules/rules_misc.xml
  <CONFIG_ITEM id="check_disable_root_account" version="1">
    <PapiOperation>
      <CHECK>
        <URI>/auth/users/root</URI>
        <BODY>{"enabled": false}</BODY>
        <KEY>users</KEY>
      </CHECK>
      <ACTION_SCOPE>NODE</ACTION_SCOPE>
      <IGNORE>FALSE</IGNORE>
      <RETRIES>0</RETRIES>
    </PapiOperation>

The hardening engine also includes a reporting component, allowing detailed reports to be generated that indicate which hardening rules are applied or not, as well as overall compliance status.  Hardening report generation can be initiated with the following CLI command syntax:

# isi hardening reports create
...............Hardening operation complete.

# isi hardening reports list
Name   Applied  Status        Creation Date            Report Age
-----------------------------------------------------------------
root-lockdown  Yes  Compliant Fri Oct 24 14:28:40 2025 3m18s
-----------------------------------------------------------------
Total: 1

A recommended practice is to configure and enable RLM in combination with the OneFS Multi-Party Approval (MPA) feature. MPA is an administrative approval mechanism that requires at least one additional trusted party to sign off on a requested change, for certain privileged actions within a PowerScale cluster. MPA helps to mitigate the risk of data loss or system configuration damage from critical actions, by vastly reducing the likelihood of accidental or malicious execution of consequential operations. As such, enabling RLM on a cluster with MPA configured means that any requested change to the RLM state will require additional approval.

The application or removal of RLM necessitates a full cluster reboot to ensure that all relevant system controls (sysctls) are properly enforced or cleared. The hardening report (‘isi hardening reports create/view’) can be used to determine which nodes require rebooting during this process.

When expanding a cluster that has RLM enabled, only nodes that have already been rebooted into Compliance Mode using the setup wizard (option 4) are permitted to join. Any new node joining an RLM-enabled cluster must be manually rebooted to inherit the RLM configuration.

In the next article in this series, we’ll explore the processes and procedures for configuring and managing RLM.

OneFS Secure Snapshots and SnapshotIQ Locks

In today’s landscape of heightened cyber threats and data breaches, the demand for immutable, tamper-resistant snapshots is growing rapidly. Often this need arises as part of a broader security mandate, ideally proactively, but oftentimes as a response to a security incident. OneFS addresses this requirement in the following ways:

On-cluster Off-cluster
·       Read-only snapshots

·       Snapshot locks

·       Role-based administration

·       Secure snapshots with multi-party authorization

 ·       SyncIQ snapshot replication

·       SmartSync snapshot replication

·       Cyber-vaulting

 

As we have seen over the course of this series of articles, the primary objective of Secure Snapshots, introduced in OneFS 9.12, is to ensure snapshot immutability, safeguarding them from accidental changes or deliberate tampering. Beyond protecting the snapshots themselves, Secure Snapshots also secures the associated snapshot schedules. Since these schedules govern the creation of future snapshots, any unauthorized modifications could lead to unintended outcomes—such as excessive snapshot generation that strains the cluster, or a complete halt in snapshot creation.

Secure Snapshots are built on the OneFS multi-party authorization (MPA) framework, which requires additional approvals before any changes can be made to snapshot configurations or schedules, effectively preventing unauthorized actions.

The Secure Snapshots functionality is in addition and complementary to traditional SnapshotIQ locking, which operates under the purview or the ‘isi snapshot locks’ CLI command set. As such, Secure Snapshots do not affect the addition, removal, or general operation of the regular SnapshotIQ snapshot locks.

Feature Availability Authorization Description
Secure snapshots OneFS 9.12 and later Multiple approvers One or more additional approvers must authorize snapshot privileged action.
SnapshotIQ locks OneFS 7.0 and later Single approver A single administrator can configure and manage snapshot locks.

While SnapshotIQ locks help prevent snapshots from being accidentally or unintentionally deleted, unlike Secure Snapshots, they do not offer the additional anti-tamper protection of Multi-party Authorization (MPA).

SnapshotIQ locks function as follows. For example, a manual snapshot, ‘snaploc1’ is taken of /ifs/test:

# isi snapshot snapshots create /ifs/test --name snaploc1

# isi snapshot snapshots list | grep snaploc1
79188 snaploc1                                     /ifs/test

A lock is then placed on it (in this case lock ID=1):

# isi snapshot locks create snaploc1

# isi snapshot locks list snaploc1
ID
----
1
----
Total: 1

Or from the WebUI:

Attempts to delete the snapshot fails because the lock prevents its removal:

# isi snapshot snapshots delete snaploc1

Are you sure? (yes/[no]): yes

Snapshot "snaploc1" can't be deleted because it is locked

The CLI command ‘isi snapshot locks delete <lock_ID>’ can be used to clear existing snapshot locks, if desired. For example,  to remove the only lock (ID=1) from snapshot ‘snaploc1’:

# isi snapshot locks list snaploc1
ID
----
1
----
Total: 1

# isi snapshot locks delete snaploc1 1
Are you sure you want to delete snapshot lock 1 from snaploc1? (yes/[no]): yes

# isi snap locks view snaploc1 1
No such lock

Or from the WebUI:

Once the lock is removed, the snapshot can then be deleted:

# isi snapshot snapshots delete snaploc1
Are you sure? (yes/[no]): yes

# isi snapshot snapshots list| grep -i snaploc1 | wc -l
       0

A snapshot can have up to a maximum of sixteen SnapshotIQ locks on it at any time. Also, lock numbers are continually incremented and not recycled upon deletion.

Similar to snapshot expiry configuration, snapshot locks can also have an expiry time configured too. For example, to set a lock on snapshot ‘snaploc1’ that expires at 12pm on April 1st, 2026:

# isi snap lock create snaploc1 --expires '2026-04-01T12:00:00'

# isi snap lock list snaploc1
ID
----
36
----
Total: 1

# isi snap lock view snaploc1 33
     ID: 36
Comment:
Expires: 2026-04-01T12:00:00
  Count: 1

Or from the WebUI:

Note that if the duration period of a particular snapshot lock expires but others remain, OneFS will not delete that snapshot until all the locks on it have been removed or expired.

Secure Snapshots Configuration and Management

As discussed in the previous article in this series, OneFS Secure Snapshots are designed to ensure the immutability of snapshots and safeguard them against accidental or malicious modification or deletion. Beyond protecting the snapshots themselves, the multi-party authorization requirement also secures the associated snapshot schedules. These schedules are critical for generating future snapshots, and any tampering – such as altering the frequency or disabling them entirely – could result in unintended consequences, including excessive snapshot creation that strains cluster resources or a complete halt in snapshot generation.

Once the cluster is committed to OneFS 9.12, snapshots are licensed, and two or more approval admins have been configured, MPA can be enabled. This preparatory work is covered in-depth in this MPA configuration article. MPA can be enabled from the WebUI under Access > Multi-Party Authorization > Settings:

Or via the CLI with the following command syntax:

# isi mpa settings global modify --enable true

The general flow of Secure Snapshot privileged command execution is as follows:

Once MPA is activated, any Secure Snapshot deletion, and other privileged action, requests are blocked pending approval.

As such, an attempt to delete a secure snapshot generates a warning, and the action is suspended until approved by an authorized user. For example:

Or from the CLI:

# isi snapshot list

ID   Name                                          Path

-----------------------------------------------------------------------------

40    Snapshot:2025July21.2:44PM                        /ifs/data

42    SIQ-8b30297763c5db7307e2d84504839c25-new      /ifs/data/zone1

# isi snapshot delete 40

Are you sure? (yes/[no]): yes

Delete Snapshot is a privileged action . A request paareqa99439a172d09999 to perform this action is pending approval. Check Multi-Party Authorization to view the status of the request.

Details of the requested snapshot privileged action can be viewed as follows:

Similarly, from the CLI:

# isi mpa requests view --id= paareqa99439a172d09999

ID: paareqa99439a172d09999

Action: delete_snapshot

Action Payload: -

Created By: MPA

Last Update Time: 2025-07-21T14:44:56

Resource IDs: 40

Resource Type: snapshot

Service: snapshot

Status: pending

Request For: 0

Assuming they sanction the privileged action, the authorizer(s) grant approval for the snapshot deletion, or other privileged action.

This is achieved by clicking the ‘Approve’ button for the pending request, under Access > Multi-Party Authorization > Requests:

The authorizer is then prompted for their TOTP security authorization code as part of the approval process:

Or via the CLI:

# isi mpa requests list

# isi mpa requests approve <id> <comment> <approved> --totp-code <******> --approval-valid-before <timestamp> --zone <zone>

Once approval has been granted, the privileged snapshot operation can complete as expected.

The WebUI reports a successful approval status as follows:

Or from the CLI:

# isi mpa requests approve --approved=true –id= paareqa99439a172d09999–comment=approved

totp_code: xxxxxx

Request paareqa99439a172d09999has been approved successfully.

# isi snapshot delete 40

Are you sure? (yes/[no]): yes

# isi snapshot list

ID   Name                                          Path

-----------------------------------------------------------------------------

42    SIQ-8b30297763c5db7307e2d84504839c25-new      /ifs/data/zone1

It’s worth noting that snapshot schedules will only generate secure snapshots if they are configured to create snaps with expiration dates. For example:

It’s also important to note that the new Secure Snapshots feature in OneFS 9.12 is designed to complement, rather than replace, the existing SnapshotIQ locking mechanism, which is managed via the ‘isi snapshot locks’ CLI command set.

Feature Availability Authorization Description
Secure snapshots OneFS 9.12 and later Multiple approvers One or more additional approvers must authorize snapshot privileged action.
SnapshotIQ locks OneFS 7.0 and later Single approver A single administrator can configure and manage snapshot locks.

Secure Snapshots operate independently and do not interfere with the creation, removal, or behavior of traditional SnapshotIQ locks. This topic will be covered in more depth in the next article in this series.

OneFS Secure Snapshots

OneFS 9.12 sees the addition of a new Secure Snapshots feature. The core ethos of this new functionality is to provide snapshot immutability, in order to protect them from being altered or deleted – either by accident or by a malicious actor. In addition to the snapshots themselves, Secure Snapshots functionality also includes protection of the snapshot schedules. Since these schedules are responsible for creating future snapshots, tampering with them could also lead to unforeseen and undesirable consequences, such as schedule modification which overwhelms the cluster with snapshots, or ceases to generate snapshots all together.

Secure snapshots are predicated upon OneFS multi-party authorization (MPA), which prevents a user from executing configuration changes on snapshots and snapshot schedules without additional approval.

Secure Snapshots in conjunction with MPA protect a subset of snapshot operations, or ‘privileged actions’. These include:

  • Deleting a snapshot prior to its expiry date
  • Shortening the expiry date of a snapshot
  • Renaming a snapshot using reserved name
  • Modifying a snapshot schedule

These following highlighted snapshot operations are privileged and require secondary approval when MPA is enabled:

Snapshot Operation MPA Required?
Deletion of a snapshot without an expiry date No
Deletion of a snapshot with an expiry date once that expiry date has passed No
Deletion of a snapshot that has an expiry date before that expiry date has passed Yes
Modifying a snapshot schedule Yes
Adding an expiry date to a snapshot No
Extending the expiry date of a snapshot No
Reducing the expiry date of a snapshot Yes
Renaming a snapshot using reserved name used Yes
Adding a lock to a snapshot No
Removing a lock from a snapshot No

With Secure Snapshots, you cannot delete or modify snapshots with expiration dates or modify a snapshot schedule without these actions going through MPA first. So when you make these requests, the information about the request is also sent to MPA so that approver can see what changes you’re trying to make so they know exactly what they are actually approving or not.

OneFS 9.12 also checks whether user-generated snapshots have overlapping names with snapshots that could be potentially created by demons on the cluster. So the basic workflow is that users will make a request to delete a snapshot or edit a snapshot and then will have to go through MPA and make the exact same request again. If the approver grants the request, the snapshot deletion will proceed. If the approver rejects or does not respond to the request, if they have approved the request, no action occurs. They’ve made the same request a second time, and if they haven’t responded or if they haven’t, if they’ve rejected the request, it just will not make the request or issue a new request.

There are three principal prerequisites for deploying Secure Snapshots, which are as follows:

  1. Secure Snapshots can only be enabled once the cluster has been committed to OneFS 9.12 or later.
  2. SnapshotIQ needs to be licensed and the ISI_PRIV_SNAPSHOT privilege is required.
  3. MPA must already be active on the cluster and configured for two or more approvers.

The MPA enablement and approver registration workflow is as follows:

1. First, Create users and grant approval privilege.

Approver registration be configured from the CLI with the following set of commands.

First, configure two approver users – in this case ‘mpa-approver1’ and ‘mpa-approver2’:

# isi auth users create mpa-approver1 --enabled true --password <passwd>

# isi auth users create mpa-approver2 --enabled true --password <passwd>

Next, create a group for these users. Eg. ‘mpa-group’:

# isi auth group create mpa-group

Add the approver users to their new group:

# isi auth group modify mpa-group --add-user mpa-approver1

# isi auth group modify mpa-group --add-user mpa-approver2

Then add the new group (mpa-group) to the RBAC ApprovalAdmin role:

# isi –debug auth roles modify ApprovalAdmin --add-group mpa-group

If needed, configure approvers’ SSH login to the cluster.

# isi auth roles create ssh-role

# isi auth roles modify ssh-role --add-group mpa-group --add-priv-read ISI_PRIV_LOGIN_SSH

Next, add the approver users to MPA and complete their registration. This involves initiating registration, using the secret or embedded URL in conjunction with a third party code generator to create a time-based one-time password or TOTP.

Here, the ‘mpa-approver1’ user is registered first:

# isi mpa initiate-registration

Initiated registration successfully: (Online utility can be used to   convert URL to QR code for use with authenticator)

account:  Dell Technologies

algorithm:  SHA1

digits:  6

issuer:  mpa-approver1

period:  30

secret:  5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI

url:  otpath://totp/Dell%Technologies:mpa-approver1?secret=5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI&issuer=Dell%20Technologies&algorithm=SHA1&digits=6&period=30

The user secret or URL returned can be used to set up a TOTP in Google Authenticator, or another similar application.

This secure one-time registration can be performed as follows:

a. First, initiate the OneFS registration process to obtain a URI, which can then be generated into a QR code by a 3rd party app. For example:

b. Next, convert the QR code (or URI) into a time-based one-time password code via Google Authenticator, Microsoft Authenticator, or another TOTP app:

c. Finally, complete the OneFS approver registration wizard:

This can be performed from the CLI as follows:

# isi mpa complete-registration

totp_code:  ******

Approver registered.

Or from the WebUI, under WebUI under Access > Multi-Party Authorization > Registration:

This process should then be repeated for the second (ie. ‘mpa-approver2’) role.

  1. Once the approvers are registered, MPA can now be enabled via its global settings.

This can be performed from the WebUI under Access > Multi-Party Authorization > Settings:

Note that MPA cannot be enabled on clusters without having at least two registered approvers.

When successfully enabled, the following status banner will be displayed:

Alternatively, enabling via the CLI can be done with the following command syntax:

# isi mpa settings global modify –enable true

# isi mpa settings global view

Multi Party Authorization is enabled

Once MPA is enabled on a cluster, any requests to delete Secure Snapshots or modify their schedules, etc, are paused pending approval. For example:

Or from the CLI:

# isi snapshot list

ID   Name                                          Path

-----------------------------------------------------------------------------

40    Snapshot:2025July21.2:44PM                        /ifs

42    SIQ-8b30297763c5db7307e2d84504839c25-new      /ifs/data/zone1

# isi snapshot delete 40

Are you sure? (yes/[no]): yes

Delete Snapshot is a privileged action . A request paareq43d57a6172a16de8 to perform this action is pending approval. Check Multi-Party Authorization to view the status of the request.

In the next article in this series, we’ll turn our attention to the configuration and management of Secure Snapshots.

Multi-party Authorization Management and Troubleshooting

OneFS Multi-party Authorization, or MPA, is an administrative approval mechanism that requires at least one additional trusted party to sign off on a requested change, for certain privileged actions within a PowerScale cluster. As such, MPA helps to reduce the likelihood of data loss or system configuration damage from critical actions, by mitigating the risk of accidental or malicious execution of consequential operations.

MPA can be managed via the OneFS WebUI, CLI, or platform API, and the WebUI portal can be found by navigating to Access > Multi-Party Authorization:

Similarly, the CLI uses ‘isi mpa’ command-set, with the following high-level options:

# isi mpa -h

Description:
    Manage Multi-Party Authorization.

Usage:
    isi mpa {<action> | <subcommand>}
        [--timeout <integer>]
        [{--help | -h}]

Actions:
    complete-registration    Complete Multi-Party Authorization registration.
    initiate-registration    Initiate Multi-Party Authorization registration.

Subcommands:
    approvers                Manage Multi-Party Authorization approvers.
    ca-certificates          Manage CA-Certificates for Multi-Party
                             Authorization.
    requests                 Manage Multi-Party Authorization requests.
    settings                 Manage Multi-Party Authorization settings.

Options:

  Display Options:
    --timeout <integer>
        Number of seconds for a command timeout (specified as 'isi --timeout NNN <command>').
    --help | -h
        Display help for this command.

See 'isi mpa <subcommand> --help' for more information on a specific subcommand.

Within the platform API (pAPI), the MPA endpoints all reside under http://<cluster_IP>:8080/platform/23/mpa/. Their hierarchy and syntax is as follows:

/23/mpa/approval/<ID>
/23/mpa/approvers
/23/mpa/approvers/<ID>
/23/mpa/complete-registration
/23/mpa/initiate-registration
/23/mpa/requests
/23/mpa/requests/<ID>
/23/mpa/settings/config/request-lifecycle
/23/mpa/settings/global
/23/mpa/settings/privilege-action/metadata
/23/mpa/signed-approval/<ID>
/23/mpa/trust-anchors

An MPA pAPI endpoint can be queried as follows:

https://<cluster_IP>:8080/platform/23/mpa/settings/global
{
"last_update_time" : -1,
"mpa_enabled" : true
}

Invalid MPA calls will return a fairly descriptive error message and code. For example, the following ‘AEC_BAD_REQUEST’ error and 400 code resulting from a request query of a nonexistent service and action:

# curl -u root:a -k 'https://<cluster_IP>:8080/platform/mpa/requests' --data '{"service":"TestService", "action":"TestAction"}'
{
"errors" :
[
{
"code" : "AEC_BAD_REQUEST",
"message" : "Invalid privileged action TestAction for service TestService"
}
]
}
Additionally, the API of choice can be ‘described’ as follows:
https://<cluster_IP>:8080/platform/23/mpa/approvers?describe
Resource URL: /platform/23/mpa/approvers

    Overview: List MPA approvers
     Methods: GET
********************************************************************************
Method GET: List MPA approvers
URL: GET /platform/23/mpa/approvers

Query arguments:
registration_status=<enum> Where <enum> is one of Registration_Initiated, Pending_Approval, Registered, Registration_Denied.

Registration status of a MPA approver.

GET response body schema:
{
  "type": [
    {
      "type": "object",
      "description": "A list of errors that may be returned.",
--<snip>--

When upgrading a cluster from an earlier release, the MPA framework’s privileged actions and approval system (PAAS) becomes available after committing the upgrade to OneFS 9.12. As part of the upgrade process, a pre-check is performed to ensure that no existing role named ‘ApprovalAdmin’ (case-insensitive) is present. If such a role exists, administrators will be prompted to rename it to avoid conflicts with the newly introduced system role. This is necessary because privileges associated with the new ApprovalAdmin role will not be applied to pre-existing roles of the same name.

Upon upgrading to OneFS 9.12, the feature upgrade flag (ERA2_MPA) is automatically set to ‘true’ when the upgrade is committed, thereby providing, but not enabling, MPA functionality. Additionally, all MPA-related APIs become accessible only after the upgrade is finalized.

It is important to note that MPA is not supported in Compliance Mode. If attempting to enable MPA on a compliance cluster, the following warning message is displayed:

# isi mpa settings global modify --enable=true

Cannot enable Multi-Party Authorization mode on clusters with compliance mode already enabled

Prior to initiating an upgrade to OneFS version 9.12, the system performs a validation check to ensure that no existing role named ApprovalAdmin—regardless of case sensitivity—is present. The Multi-Party Authorization (MPA) feature becomes available only after the upgrade has been successfully committed. MPA does not require any special licensing; however, for enhanced security, it is recommended to enable MPA in conjunction with Root Lockdown Mode (RLM), which is also introduced in OneFS 9.12. Activation of RLM requires that the compadmin user is enabled and that both the Hardening and SmartLock licenses are valid and active across the cluster.

Note that the ability to disable MPA is not supported in OneFS 9.12, so consider this before activating the MPA feature.

When troubleshooting MPA-related issues, particularly those involving Time-Based One-Time Passwords (TOTP), several factors should be considered. For example:

If a ‘TOTP is invalid’ error occurs during the approver registration process, verify that the TOTP code is being generated from the correct account in the third-party application. Time synchronization between the client and the cluster is critical, and any time skew can result in token validation failures. If an approver loses their TOTP token or secret key, they must reinitiate registration. When MPA is enabled, this re-registration becomes a privileged action and requires approval.

In a rare scenario where all the MPA approvers have lost access to their TOTP credentials and no one is available to approve privileged actions, OneFS does provide a last resort ‘break-glass’ mechanism to handle such cases. Where appropriate, Dell can issue a specific patch that allows the approval of a registration request for a new ApprovalAdmin. This patch must be executed by the root user and will internally run the ‘isi mpa request approve’ CLI command for the specific request ID. This bypasses the TOTP check and is limited to the ‘register_approval_admin’ action. If such a scenario is encountered, please contact Dell Support immediately.

An MPA approval request can be in one of five states: Approved, Cancelled, Completed, Pending, or Rejected.

These states, and their various expiry times, can be viewed with the following CLI command syntax:

# isi mpa settings request-lifecycle list

Approved
    Description: expire in configured days for privilege action execution.
    Expire_time: 7d

Cancelled
    Description: remove from system in configured days.
    Expire_time: 7d

Completed
    Description: remove from system in configured days.
    Expire_time: 30d

Pending
    Description: expire in configured days can be updated by user or approver.
    Expire_time: 7d

Rejected
    Description: remove from system in configured days.
    Expire_time: 30d

The MPA requests comprise the following attributes.
Request Attribute Description
action Name of privileged action
action_payload Set of key/value pairs for privileged action payload
cluster_guid Cluster GUID that uniquely identifies PowerScale
created_by Specified who created the request
creation_time Creation time of the request. Unix epoch time format
id Unique ID of MPA request.
last_update_time Last update time of the request.  Unix epoch time format
request_for User for whom request is created for.
resource_ids List of resources IDs requested for approval.
resource_type Type of resource requested for approval.
service Name of service or component in system that owns the privileged action
status Status of MPA request
system_created A privileged action approval request was auto created by the system or not.
zone_id Zone id of MPA request created.

If a privileged action fails to execute despite an approved request, it is necessary to verify whether the approval has expired. For troubleshooting purposes, diagnostic logs related to MPA can be found in the PAPI logs (/var/log/isi_papi_d.log). A fair amount of detail and context is captured by default without the need to increase logging verbosity. For example, the following secure snaphshot delete request, which includes snap ID, warning message, MPA request ID, URI info, error codes, etc:

2025-09-21T21:45:09.734523+00:00 <3.6> GLaDOS-8(id8) isi_papi_d[46945]: STACK snapshot ID: 2, Delete Snapshot is a privileged action. A request paareq43d57a6172a16de8 to perform this action is pending approval. Check Multi-Party Authorization to view status of the request.     from --- (---:0):      isi_exception::isi_exception(int, char const*, __va_list_tag*) (OFFSET:134)     api_exception::api_exception(api_error_code, char const*, ...) (OFFSET:146)     snapshot_mpa_interface::initiate_mpa_request(mpa_lib::MpaMatchContext&, std::__1::basic_string<char, mpa_lib::MpaMatchContext&::char_traits<char>, mpa_lib::MpaMatchContext&::allocator<char> > const&) (OFFSET:281)     snapshot_snapshot_handler::http_delete(request const&, response&) (OFFSET:2136)     uri_handler::execute_http_method_internal(request&, response&) (OFFSET:253)     uri_handler::execute_http_method(request&, response&, bool, bool, bool) (OFFSET:1894)     uri_manager::execute_request(request&, response&, bool, bool, bool, bool, bool, bool) (OFFSET:1578)     std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf(void) (OFFSET:7452)     std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf(void) (OFFSET:11172)     typeinfo name for api_exception (OFFSET:16919)     typeinfo name for api_exception (OFFSET:13861)     ADDRESS (UNKNOWN:-1806659576)

The OneFS services which include MPA privileged actions can be queried with the following CLI syntax:

# isi mpa settings pa-metadata list

For example:

For these MPA-integrated services, such as S3, service-specific logs can also provide useful additional diagnostics information. These MPA privileged actions and their associated logfiles in OneFS 9.12 include:

Service /

Component

 Privileged Action   Logfile
Hardening apply_hardening

disable_hardening

 /var/log/hardening.log

/var/log/hardening_engine.log
MPA register_approval_admin

upload_trust_anchor

 /var/log/isi_papi_d.log
S3 reduce_immutable_bucket_retention

modify_server_access_logging_config

 /var/log/s3.log
Snapshot delete_snapshot

modify_snapshot

delete_snapshot_schedule

modify_snapshot_schedule

 /var/log/isi_snapshot_d.log

Complementary to MPA, OneFS 9.12 also introduces Root Lockdown Mode (RLM), which disables and blocks access to a cluster’s root account, forcing the use of roles-based access controls (RBAC). Since the root has explicit permission to bypass security controls, locking it down can also help to prevent a single bad actor from performing destructive operations on the data or storage platform.

As a best practice, enabling Root Lockdown Mode alongside Multi-party Authorization (MPA) provides a more secure and resilient configuration, ensuring full benefits of the MPA framework. To activate RLM, the compadmin user must be enabled, and both Hardening and SmartLock licenses are required. MPA and RLM can be enabled independently, and once MPA is activated, enabling RLM is a privileged action which requires an approval. RLM will be covered in-depth in a near-future article.

For disaster recovery in environments where both RLM and MPA are enabled, two procedures are available for recovering access to a node with a lost root password. After first contacting Dell Support, the preferred method involves smart-failing the node, re-imaging it, and rejoining it to the cluster. If this is not feasible, an alternative approach is to boot the node from alternate media and manually edit the System file provider to restore root access. After recovery, the Configurable Hardening Engine report should be reviewed to ensure all RLM rules are applied. If not, follow the documented steps to re-enable RLM.

To ensure the one-time usage of TOTP tokens, OneFS stores both the token and its generation counter in the MPA database under the approver’s record. When a new TOTP validation request is received, the system checks whether the token matches the previously used one. If it does, an error is triggered. The counter used to generate the token must be higher than the last used counter, ensuring that even unused older tokens cannot be reused once a newer token has been validated.

For example, if a token is generated at 12:00:00 and submitted at 12:00:20, the MPA uses the current time window (12:00:00–12:00:30) to generate the counter and validate the token. Once validated, both the token and counter are stored. If a malicious actor attempts to reuse the token, validation will fail either because the token matches the last used one or because the counter is outdated.

OneFS Multi-party Authorization Configuration

The previous article in this series looked at the OneFS Multi-party Authorization architecture, which requires that one or more additional trusted parties sign off on requested changes for certain privileged actions within a PowerScale cluster.

At a high-level, the MPA workflow is as follows:

  1. The first step is registration, which requires two users with the approval privilege, both registered as internal approvers. And note that merely having an RBAC approval privilege is not enough – there is a secure process to register the users. Once there are two registered approval users, then it’s on to step #2, where the approval admin can enable the MPA feature.
  2. Once MPA is enabled, any of the predefined ‘privileged actions’ will require explicit approval from internal approvers.
  3. Step three involves generating a Privileged Action Approval Request (PAA) – either manually, or automatically when a privileged action is attempted.
  4. Next, one of the registered MPA approvers blesses or rejects the PAA Request.
  5. And finally, assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

In order to approve requests, an approver needs to complete a secure one-time registration by entering the six-digit TOTP code, etc.

MPA can be configured and managed via the OneFS CLI, platform API, or WebUI. OneFS 9.12 sees the addition of a new ‘isi mpa’ CLI command set who’s basic syntax is as follows:

# isi mpa
Description:
    Manage Multi-Party Authorization.

Usage:
    isi mpa {<action> | <subcommand>}
        [--timeout <integer>]
        [{--help | -h}]

Actions:
    complete-registration    Complete Multi-Party Authorization registration.
    initiate-registration    Initiate Multi-Party Authorization registration.

Subcommands:
    approvers                Manage Multi-Party Authorization approvers.
    ca-certificates          Manage CA-Certificates for Multi-Party
                             Authorization.
    requests                 Manage Multi-Party Authorization requests.
    settings                 Manage Multi-Party Authorization settings.

For example, the CLI syntax to view the MPA global settings:

# isi mpa settings global view

Multi Party Authorization is enabled

Similarly, the platform API MPA endpoints reside under ‘/platform/23/mpa’. For example, the endpoint to get the MPA global settings:

https://<cluster_ip_addr>:8080/platform/23/mpa/settings/global

{

"last_update_time" : 1753132456,

"mpa_enabled" : true

}

The WebUI configuration and management portal for MPA is located at Access > Multi-Party Authorization. For example, navigating to Access > Multi-Party Authorization will yield the MPA global settings:

The CLI commands and platform API endpoints for registration, query, and approval are as follows:

MPA Action CLI Command Platform API Endpoint
Initiate (re)registration isi mpa initiate-registration (–force) POST /platform/23/mpa/initiate-registration (?force)
Complete registration isi mpa complete-registration –totp_code xxxxxx PUT /platform/23/mpa/complete-registration -d ‘{“totp_code“ : xxxxxx}’
List all approvers isi mpa approver list GET /platform/23/mpa/approvers
Query an approver isi mpa approver view <ID> GET /platform/23/mpa/approvers/<ID>
Approve a request isi mpa requests approve <id> –totp xxxxxx –approved true POST /platform/23/mpa/approval/<MPA-req-id>

Note that the ability to disable multi-party authorization is not supported in OneFS 9.12, so consider this carefully before activating the MPA feature.

The five basic steps for the MPA configuration and management are as follows:

  1. First, Create users and grant approval privilege.

The approver registration and enablement workflow is as follows:

Approver registration be configured from the CLI with the following set of commands.

First, configure two approver users – in this case ‘mpa-approver1’ and ‘mpa-approver2’:

# isi auth users create mpa-approver1 --enabled true --password <passwd>

# isi auth users create mpa-approver2 --enabled true --password <passwd>

Next, create a group for these users. Eg. ‘mpa-group’:

# isi auth group create mpa-group

Add the approver users to their new group:

# isi auth group modify mpa-group --add-user mpa-approver1

# isi auth group modify mpa-group --add-user mpa-approver2

Then add the new group (mpa-group) to the RBAC ApprovalAdmin role:

# isi –-debug auth roles modify ApprovalAdmin --add-group mpa-group

If needed, configure approvers’ SSH login to the cluster.

# isi auth roles create ssh-role

# isi auth roles modify ssh-role --add-group mpa-group --add-priv-read ISI_PRIV_LOGIN_SSH

Next, add the approver users to MPA and complete their registration. This involves the approver user logging in to their account and initiating registration, using the secret or embedded URL in conjunction with a third party code generator to create a time-based one-time password or TOTP.

Here, the ‘mpa-approver1’ user is registering first:

# isi mpa initiate-registration

   Initiated registration successfully: (Online utility can be used to   convert URL to QR code for use with authenticator)

   account:  Dell Technologies

   algorithm:  SHA1

   digits:  6

   issuer:  mpa-approver1

   period:  30

   secret:  5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI

   url:  otpath://totp/Dell%Technologies:mpa-approver1?secret=5KHIHISJZPB7UTPRNEBLSIMI5L4R6K6UI&issuer=Dell%20Technologies&algorithm=SHA1&digits=6&period=30

The user secret or URL returned can be used to set up a TOTP in Google Authenticator, or another similar application.

Specifically, this secure one-time registration involves:

a. First, initiating the OneFS registration process to obtain a URI, which can then be generated into a QR code by a 3rd party app if desired. For example:

b. Next, converting the QR code or URI into a time-based one-time password code via Google Authenticator, Microsoft Authenticator, or another TOTP app:

c. Finally, completing the OneFS approver registration wizard:

Or from the CLI:

# isi mpa complete-registration

totp_code:  ******

Approver registered.

This process should then be repeated for the second (ie. ‘mpa-approver2’) role.

  1. Once the approvers are registered, MPA can now be enabled via its global settings.

MPA can be enabled from the WebUI under Access > Multi-Party Authorization > Settings:

Note that MPA cannot be enabled on clusters without having at least two registered approvers:

When successfully enabled, the following status banner will be displayed: 

Alternatively, enabling via the CLI can be done with the following command syntax:

# isi mpa settings global modify –enable true

# isi mpa settings global view

Multi Party Authorization is enabled

 

  1. Now MPA is up and running, privileged action requests can be generated.

We’ll use the new OneFS 9.12 Secure Snapshots feature as an example.

Once MPA is enabled on a cluster, any requests to execute a privileged request are paused pending approval.

In this case, an attempt to delete a secure snapshot results in a warning, and the requested action is suspended until approved by an authorizing party.

  1. Request Approval.

The privileged action requests can now be approved or rejected, as appropriate.

To private consent for the operation, the authorizing administrator clicks the ‘Approve’ button for the pending request, located under Access > Multi-Party Authorization > Requests:

The authorizing party is prompted for their time-based one-time password (TOTP) security authentication code as part of the approval process:

Or via the CLI:

# isi mpa requests list

# isi mpa requests approve <id> <comment> <approved> --totp-code <******> --approval-valid-before <timestamp> --zone <zone>
  1. Finally, the privileged action is ready to be executed.

Once approval has been granted, the WebUI reports a successful approval status.

The privileged secure snapshot delete operation is now able to proceed as expected:

Or from the CLI:

# isi mpa requests view <id>

In the next and final article in this series, we’ll turn our attention to MPA management and troubleshooting.

OneFS Multi-party Authorization Architecture

As we saw in the previous article in this series, OneFS Multi-party Authorization (MPA) requires that one or more additional trusted parties sign off on requested changes for certain privileged actions within a PowerScale cluster.

A prerequisite to enabling MPA in OneFS 9.12 is the requirement for two users with the approval privilege, both registered as internal approvers. Once MPA has been activated on a cluster, any of the predefined ‘privileged actions’ will require explicit approval. In order for a privileged action, such as secure snapshot deletion, to complete, one of the registered MPA approvers must bless the PAA request. Finally, assuming approval was granted, the privileged action can be executed.

The MPA privileged action approval workflow can be summarized in the following request state diagram:

Under the hood, MPA request lifecycle configuration data is stored in the form of key-value pairs, and the MPA Approvers registration status, PAA requests, and request approval details are kept in a shared SQLite database.

Note that MPA approver users can only reside in the System Zone and require the new ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST privileges at a minimum.

MPA includes privileged actions, which extend across the S3 protocol, supporting immutable bucket retention and logging, cluster hardening, and the main one which is the new secure snapshots functionality.

As such, the full complement of these privileged actions in OneFS 9.12 is as follows:

Service /

Component

 Action Description 
MPA register_approval_admin Registering approval admin becomes a privilege action once MPA is enabled.
MPA upload_trust_anchor Upload a trust anchor certificate after MPA is enabled.
Privileged CLI cli_mpa_check MPA check to execute specific CLI command. Eg. once root lockdown mode is enabled, elevate to root will use this.
Hardening apply_hardening Applying a Hardening Profile, such as root-lockdown, STIG.
Hardening disable_hardening Disabling a Hardening Profile, such as root-lockdown, STIG.
S3 reduce_immutable_bucket_retention Reduce bucket retention for an immutable bucket.
S3 modify_server_access_logging_config Change a bucket’s access logging configuration.
Platform reduce_immutable_bucket_retention Reduce bucket retention for an immutable bucket.
Platform modify_server_access_logging_config Changing a bucket’s access logging configuration.
Snapshot delete_snapshot Delete a snapshot.
modify_snapshot Modify a snapshot.
delete_snapshot_schedule Delete a snapshot’s schedule.
modify_snapshot_schedule Modify a snapshot’s schedule.

A number of new RBAC privileges are introduced in 9.12 to support MPA, including MPA request, MPA approval, which allows an approver to bless or reject a privileged action request, plus MPA upload, global and access zone settings:

MPA Privilege Description Permission
ISI_PRIV_MPA_APPROVAL Privilege for MPA approver to approve or reject MPA request. Write
ISI_PRIV_MPA_REQUEST Privilege required for MPA request APIs. Write/Read
ISI_PRIV_MPA_SETTINGS_GLOBAL Privilege required to enable MPA. Write
ISI_PRIV_MPA_SETTINGS_ZONE Privilege required to read MPA metadata/ MPA request-lifecycle settings. Read
ISI_PRIV_SIGNED_APPROVAL_UPLOAD Upload signed approval for Multi-Party Authorization Requests. (Applicable for 3rd party MPA only) Write

There’s also the new ‘Approval Admin’ default role, and a number of other roles that see the addition of MPA privileges, including SecurityAdmin, SystemAdmin, zone admin, and basic user, etc.

The ApprovalAdmin role is automatically assigned all the required privileges for MPA approval and configuration. For example:

# isi auth roles list | grep -i appr

ApprovalAdmin

# isi auth roles view ApprovalAdmin
       Name: ApprovalAdmin
Description: Allows MPA request approval.
    Members: root
             admin
 Privileges
             ID: ISI_PRIV_LOGIN_PAPI
     Permission: +

             ID: ISI_PRIV_MPA_APPROVAL
     Permission: w

             ID: ISI_PRIV_MPA_REQUEST
     Permission: w

             ID: ISI_PRIV_MPA_SETTINGS_GLOBAL
     Permission: r

             ID: ISI_PRIV_MPA_SETTINGS_ZONE
     Permission: r

The minimum MPA privileges required are ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST.

Here are the system access zone roles that see the addition of MPA privileges in OneFS 9.12:

System Zone Roles Privilege Permission
ApprovalAdmin ISI_PRIV_MPA_APPROVAL Write
ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_GLOBAL Read
ISI_PRIV_MPA_SETTINGS_ZONE Read
BasicUserRole ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_ZONE Read
SecurityAdmin ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_GLOBAL Write
ISI_PRIV_MPA_SETTINGS_ZONE Read
SystemAdmin ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_GLOBAL Read
ISI_PRIV_MPA_SETTINGS_ZONE Read

Similarly, the OneFS 9.12 MPA privileges for non-system zone roles:

Non-System Zone Roles Privilege Permission
BasicUserRole ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_ZONE Read
ZoneAdmin ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_ZONE Read
ZoneSecurityAdmin ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_ZONE Read

MPA approvers can use any of a cluster’s authentication providers, such as file, local, Active Directory or LDAP, and can they grant or reject any MPA request from any zone. However, an approver cannot approve their own request, regardless of whether it’s created by them or created for them.

The CLI commands and platform API endpoints for registration, query, and approval are as follows:

MPA Action CLI Command Platform API Endpoint
Initiate (re)registration isi mpa initiate-registration (–force) POST /mpa/initiate-registration (?force)
Complete registration isi mpa complete-registration –totp_code xxxxxx PUT /mpa/complete-registration -d ‘{“totp_code“ : xxxxxx}’
List all approvers isi mpa approver list GET /mpa/approvers
Query an approver isi mpa approver view <ID> GET /mpa/approvers/<ID>
Approve a request isi mpa requests approve <id> –totp xxxxxx –approved true POST /mpa/approval/<MPA-req-id>

Note that the approver user can easily re-register at any time, say if their TOTP is lost, or for another security concern.

An approval request can be in one of five states: Approved, Cancelled, Completed, Pending, or Rejected.

These states, and their various expiry times, can be viewed with the following CLI command syntax:

# isi mpa settings request-lifecycle list

Approved
    Description: expire in configured days for privilege action execution.
    Expire_time: 7d

Cancelled
    Description: remove from system in configured days.
    Expire_time: 7d

Completed
    Description: remove from system in configured days.
    Expire_time: 30d

Pending
    Description: expire in configured days can be updated by user or approver.
    Expire_time: 7d

Rejected

    Description: remove from system in configured days.
    Expire_time: 30d

The next article in this series will focus on the specific configuration, enablement, and approval procedures for MPA.

OneFS Multi-party Authorization

Among a handful of new security features, OneFS 9.12 introduces Multi-Party Authorization (MPA). MPA is an administrative approval mechanism that requires at least one additional trusted party to sign off on a requested change, for certain privileged actions within a PowerScale cluster.

As such, MPA helps to mitigate the risk of data loss or system configuration damage from critical actions, by vastly reducing the likelihood of accidental or malicious execution of consequential operations. Conceptually, MPA is fairly analogous to a bank’s safe deposit box, which requires dual control: Both the authorized owner and a bank official must be present in order to open the vault and unlock the box, ensuring that no single party can access its contents independently (dual authorization). MPA enforces a similar security control mechanism wherein operations involving critical or sensitive systems or data require explicit approval from multiple authorized entities. This ensures that no single actor can execute high-impact actions unilaterally, thereby mitigating risk and enhancing operational integrity through enforced oversight.

As such, many enterprises require MPA in order to meet industry data protection requirements, in addition to internal security mandates and best practices.

MPA can be configured and managed from the CLI, WebUI or platform API, but note that the feature is not activated by default. Once MPA has been enabled and configured, all predefined ’privileged actions’ require additional approval from an authorizing user. So this is markedly different from earlier OneFS releases, where the original requester would likely have had the basic rights and been able to execute that action in its entirety themselves.

MPA provides an approval mechanism for a set of ‘privileged actions’, which are defined in OneFS. Additionally, a new default ‘approval admin’ RBAC role is added in OneFS 9.12 specifically for this purpose. This ApprovalAdmin role is automatically assigned all the required privileges for MPA approval and configuration. Specifically:

System Zone Roles Privilege Permission
ApprovalAdmin ISI_PRIV_MPA_APPROVAL Write
ISI_PRIV_MPA_REQUEST Write
ISI_PRIV_MPA_SETTINGS_GLOBAL Read
ISI_PRIV_MPA_SETTINGS_ZONE Read

There are no special licensing requirements for enabling or running MPA. So, upon installation or upgrade to OneFS 9.12, once two or more approval users have been registered, the MPA feature itself can be enabled:

Note that MPA is incompatible with, and therefore not supported on, a PowerScale cluster that is already running in Compliance Mode.

Once MPA is up and running, any approval that’s granted for a privileged operation will remain valid and viable for a defined duration. When approving a request, the approver can specify an expiry time. Once that allotted time has passed, the approved request can no longer be used to execute an action, and a new approval must be generated. If the approver does not specify an expiry time, then, by default, their approval will be valid for seven days.

At a high-level, the MPA workflow is as follows:

  1. The first step is registration, which requires two users with the approval privilege, both registered as internal approvers. And note that merely having an RBAC approval privilege is not enough – there is a secure process to register the users. Once there are two registered approval users, the approval admin can enable the MPA feature.

    With MPA is enabled, Privileged Action Approval Requests (PAA) can be generated – either manually, or automatically when a privileged action is attempted. Any of the predefined ‘privileged actions’ will require explicit approval from internal approvers. Next, one of the registered MPA approvers blesses or rejects the PAA Request. And finally, assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

    Step Action Description
    1 Approver Registration Registration requires two users with the approval privilege, both registered as internal approvers. MPA requires a secure process to register the users.
    2 MPA Enablement Enable MPA via the CLI, WebUI, or platform API. Any of the predefined ‘privileged actions’ will require explicit approval from internal approvers.
    3 Request Creation Generate a Privileged Action Approval Request (PAA) – either manually, or automatically when a privileged action is attempted.
    4 Request Approval One of the registered MPA approvers blesses or rejects the PAA request.
    5 Request Execution Assuming approval was granted, the requesting administrator can now go ahead and execute their privileged action.

MPA approver users can only reside in the System Zone, and require the new ISI_PRIV_MPA_APPROVAL and ISI_PRIV_MPA_REQUEST privileges at a minimum. So while a request can be generated from any zone, an approver for that request must reside in the system zone.

An approval request can be in one of the following states:

Approval Request State Description Expiry Time
Approved Indicates request is approved by an approver. Will expire in 7 days by default, or at approver-configured expiry time. 7 days
Cancelled Indicates request is Cancelled by user who created the request. will be removed from system in 7 days. 7 days
Completed Approved request has been executed, and cannot be used again. Will be removed from system in 30 days. 30 days
Pending Awaiting for approval, prior to expiry in 7 days. 7 days
Rejected Indicates request is rejected by approver, and will be removed from system in 30 days. 30 days

Note that, in order to approve or reject requests, an approver needs to complete a secure one-time registration.

In OneFS 9.12, the MPA request lifecycle configuration is predefined, and no updates are supported. Only the Approval API (approval_valid_before) can shorten MPA request expiration time in approved status. MPA requests are zone scoped and managed by MPA request lifecycle configuration. An authorized user can view or update a request while it’s in a ‘pending’ state, but only an MPA approver user can move the request to an ‘approved’ or ‘rejected’ state. While an approval is valid (i.e. not expired), the original requester can use it to execute the privileged action. A request stays in an ‘approved’ state until its approval expires, at which point it is moved to a ‘completed’ state.

MPA includes privileged actions, which extend across the S3 protocol, supporting immutable bucket retention and logging, cluster hardening, and the main one which is the new secure snapshots functionality. These are available for selection in the MPA configuration, for example from the WebUI MPA Requests dropdown menu:

In the next article in this series, we’ll look closer at the architectural fundaments that constitute MPA.

PowerScale Cybersecurity Suite

The Dell PowerScale Cybersecurity Suite represents a forward-thinking, integrated approach to addressing the growing challenges in cybersecurity and disaster recovery.

It aligns with Gartner’s definition of Cyberstorage, a category of solutions specifically designed to secure data storage systems against modern threats. These threats include ransomware, data encryption attacks, and theft, and the emphasis of Cyberstorage is on active prevention, early detection, and the ability to block attacks before they cause damage. Recovery capabilities are also tailored to the unique demands of data storage environments, making the protection of data itself a central layer of defense.

Unlike traditional solutions that often prioritize post-incident recovery – leaving organizations exposed during the critical early stages of an attack – Dell’s PowerScale Cybersecurity Suite embraces the Cyberstorage paradigm by offering a comprehensive set of capabilities that proactively defend data and ensure operational resilience. These include:

Capability Details
Active Defense at the Data Layer The suite integrates AI-driven tools to detect and respond to threats in real-time, analyzing user behavior and unauthorized data access attempts. Bidirectional threat intelligence integrates seamlessly into SIEM, SOAR, and XDR platforms for coordinated protection at every layer.
Automated Threat Prevention Includes features like automated snapshots, operational air-gapped vaults, and immediate user lockouts to stop attacks before damage spreads. Attack simulation tools also ensure that the defenses are always optimized for emerging threats.
NIST Framework Alignment Adheres to the National Institute of Standards and Technology (NIST) cybersecurity framework, providing a structured approach to identifying, protecting, detecting, responding to, and recovering from threats. This comprehensive protection eliminates vulnerabilities overlooked by traditional backup and security tools, enabling organizations to stay ahead of today’s evolving cyber risks while ensuring business continuity.
Rapid Recovery and Resilience With secure backups and precision recovery, organizations can rapidly restore specific files or entire datasets without losing unaffected data. Recovery is accelerated by integrated workflows that minimize downtime.

By embedding detection, protection, and response directly into the data layer, the Dell PowerScale Cybersecurity Suite adopts a proactive and preventive approach to safeguarding enterprise environments:

Approach Details
Identification & Detection Detecting potential incursions in real time using AI-driven behavioral analytics.
Protection Protecting data at its source with advanced security measures and automated threat monitoring.
Response Responding decisively with automated remediation to minimize damage and accelerate recovery, ensuring seamless continuity.
Recovery Providing recovery tools and forensic data and recovery tools to quickly restore clean data in the event of a breach, minimizing business disruption.

It begins with identification and detection, where data is protected at its source through advanced security measures and continuous automated threat monitoring. Protection is achieved by identifying potential incursions in real time using AI-driven behavioral analytics, allowing organizations to act before threats escalate. When a threat is detected, the suite responds with automated remediation processes that minimize damage and accelerate recovery, ensuring uninterrupted operations.

This integrated approach enables Dell to address the full lifecycle of security and recovery within the PowerScale platform, delivering exceptional resilience across IT environments.

Released globally on August 28, 2025, the Dell PowerScale Cybersecurity Suite is available in three customizable bundles tailored to meet diverse operational and regulatory needs.

Bundle Details
Cybersecurity Bundle Leverages AI-driven threat detection, Zero Trust architecture, and automated risk mitigation to fortify security.
Airgap Vault Bundle Extends the capabilities of the Cybersecurity Bundle by adding isolated, secure backups for robust ransomware protection. This bundle requires the Cybersecurity Bundle and the Disaster Recovery Bundle..
Disaster Recovery Bundle Prioritizes rapid recovery with near-zero Recovery Point Objectives (RPO), Recovery Time Objectives (RTO), and seamless failover capabilities.

The Cybersecurity Bundle leverages AI-driven threat detection, Zero Trust architecture, and automated risk mitigation to strengthen data protection. The Airgap Vault Bundle builds on this by adding isolated, secure backups for enhanced ransomware defense, and requires both the Cybersecurity and Disaster Recovery bundles for full deployment. The Disaster Recovery Bundle focuses on rapid recovery, offering rapid Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO), along with seamless failover capabilities to ensure business continuity.

Customers can choose specific bundles based on their requirements, with the only prerequisite being that the Airgap Vault bundle must be deployed alongside both the Cybersecurity and Disaster Recovery bundles to ensure full functionality and integration.

Built upon the venerable PowerScale platform, the suite is engineered to protect unstructured data and maintain operational availability in today’s increasingly complex threat landscape. As such, it offers a comprehensive set of tools, techniques, and architectural flexibility to deliver multilayered security and responsive recovery.

This enables organizations to design robust solutions that align with their specific security priorities—from advanced threat detection to seamless disaster recovery.

Among its key benefits, the suite includes automated threat response capabilities that swiftly mitigate risks such as malicious data encryption and exfiltration.

Feature Details
Automated threat response The suite features automated responses to cybersecurity threats, such as data encryption, and the prevention of data exfiltration, helping mitigate risks swiftly and effectively.
Secure Operational Airgap Vault Data is protected within an isolated operational airgap vault, and will only be transferred to the operational airgap vault if the production storage environment is not under attack, ensuring critical assets remain secure and inaccessible to unauthorized actors.
Ecosystems Integration Seamlessly integrates with leading endpoint protection and incident response software, automating and simplifying operations during a cyberattack to ensure a coordinated and efficient response.
DoD-Certified Hardware Integration Designed to enhance PowerScale’s DoD APL certified hardware, meeting rigorous cybersecurity standards, and providing customers with a trusted platform on which to build their defenses. The suite’s advanced capabilities, robust protection, and proven hardware deliver a comprehensive cyber and DR solution tailored to meet today’s complex security challenges.

It also features a secure operational airgap vault, which ensures that data is only transferred when the production environment is verified to be safe, keeping critical assets isolated and protected from unauthorized access. Integration with leading endpoint protection and incident response platforms allows for coordinated and efficient responses during cyberattacks, streamlining operations and reducing complexity.

The suite is also designed to complement PowerScale’s DoD APL-certified hardware, meeting stringent cybersecurity standards and providing a trusted foundation for enterprise defense strategies. Its advanced capabilities, combined with proven hardware, deliver a comprehensive cybersecurity and disaster recovery solution tailored to modern security challenges.

The Dell PowerScale Cybersecurity Suite is engineered to support petabyte-scale environments containing billions of files distributed across multiple PowerScale clusters. There is no hard-coded limit on the volume of data it can manage. However, actual throughput and failover times are influenced by factors such as SyncIQ bandwidth, the degree of policy parallelism, and the overall readiness of the environment. To help forecast recovery timelines, the suite includes an Estimated Failover Time engine that uses real metrics to generate policy-specific projections.

Superna software, which underpins the suite, interacts with PowerScale systems via REST APIs. These APIs provide access to inventory data, facilitate the orchestration of snapshot creation and deletion, and enable user lockout and restoration of access to shared resources. Specifically, the suite utilizes the file system API and the system configuration API to perform these operations. Meanwhile, configuration and management can be performed via the comprehensive, intuitive WebUI:

The performance impact of running the Data Security Edition—which includes the Cybersecurity Bundle with Ransomware Defender and Easy Auditor—is primarily tied to the frequency and volume of API calls made to PowerScale. These calls can be managed and tuned by administrators and support teams during deployment and system configuration. One known scenario that may affect cluster performance occurs when user permissions are broadly configured to allow all users access to all shares. In such cases, default settings combined with enabled snapshots can lead to excessive API endpoint activity, as the system attempts to create and delete snapshots across all shares. To mitigate this, Dell recommends disabling default snapshots during deployment and instead configuring snapshots only for critical data paths. Outside of this specific configuration, Dell has not identified any significant performance impacts under normal operating conditions.

The Dell-branded software is built on the same codebase as the standard Superna 2.x release, with branding determined by licensing. This branding helps  ensure consistency, simplify user interactions, and reinforce the alignment between the suite and Dell Technologies’ broader product portfolio.

With regard to release cadence, there is typically a 60-day lead time between Superna releasing a new version and Dell launching its branded equivalent, allowing for additional QA, regression, and longevity testing.

With the Cybersecurity suite, Dell Technologies will directly manage implementation services and provide first-call support. This approach ensures a seamless and customer-focused experience, with faster response times and streamlined service delivery. It also guarantees that customers receive consistent and integrated support throughout their deployment and operational lifecycle.

It is important to note that Dell-branded and Superna-branded software cannot be mixed within the same PowerScale cluster. Currently, the Dell PowerScale Cybersecurity Suite is intended exclusively for new deployments and is not compatible with existing Superna-branded environments. Migration from Superna to Dell-branded software is not supported at this time, unless an entire Superna Eyeglass solution is being renewed. However, Dell is actively working to expand migration options in future releases.

PowerScale InsightIQ 6.1

It’s been a sizzling summer for Dell PowerScale to date! Hot on the heels of the OneFS 9.12 launch comes the unveiling of the innovative new PowerScale InsightIQ 6.1 release.

InsightIQ provides powerful performance and health monitoring and reporting functionality, helping to maximize PowerScale cluster efficiency. This includes advanced analytics to optimize applications, correlate cluster events, and the ability to accurately forecast future storage needs.

So what new goodness does this InsightIQ 6.1 release add to the PowerScale metrics and monitoring mix?

Additional functionality includes:

Feature New IIQ 6.1 Functionality
Ecosystem support ·         InsightIQ is qualified on Ubuntu
Flexible Alerting ·         Defining custom alerts on the most used set of granular metrics.

·         Nine new KPIs for a total of 16 KPIs.

·         Increased granularity of alerting and many more
Online Migration from Simple to Scale ·         Customers with InsightIQ 6.0.1 Simple (OVA) can now migrate data and functionalities to InsightIQ 6.1.0 Scale.
Self-service Admin Password Reset ·         Administrators can reset their own password through a simple, secure flow with reduced IT dependency

InsightIQ 6.1 continues to offer the same two deployment models as its predecessors:

Deployment Model Description
InsightIQ Scale Resides on bare-metal Linux hardware or virtual machine.
InsightIQ Simple Deploys on a VMware hypervisor (OVA).

The InsightIQ Scale version resides on bare-metal Linux hardware or virtual machine, whereas InsightIQ Simple deploys via OVA on a VMware hypervisor.

InsightIQ v6.x Scale enjoys a substantial breadth-of-monitoring scope, with the ability to encompass 504 nodes across up to 20 clusters.

Additionally, InsightIQ v6.x Scale can be deployed on a single Linux host. This is in stark contrast to InsightIQ 5’s requirements for a three Linux node minimum installation platform.

Deployment:

The deployment options and hardware requirements for installing and running InsightIQ 6.x are as follows:

Attribute InsightIQ 6.1 Simple InsightIQ 6.1 Scale
Scalability Up to 10 clusters or 252 nodes Up to 20 clusters or 504 nodes
Deployment On VMware, using OVA template RHEL, SLES, or Ubuntu with deployment script
Hardware requirements VMware v15 or higher:

·         CPU: 8 vCPU

·         Memory: 16GB

·         Storage: 1.5TB (thin provisioned);

Or 500GB on NFS server datastore

 Up to 10 clusters and 252 nodes:

·         CPU: 8 vCPU or Cores

·         Memory: 16GB

·         Storage: 500GB

Up to 20 clusters and 504 nodes:

·         CPU: 12 vCPU or Cores

·         Memory: 32GB

·         Storage: 1TB
Networking requirements 1 static IP on the PowerScale cluster’s subnet 1 static IP on the PowerScale cluster’s subnet

Ecosystem support:

The InsightIQ ecosystem itself is also expanded in version 6.1 to also include Ubuntu 24.04 Online deployment and OpenStack RHOSP 21 with RHEL 9.6, in addition to SLES 15 SP4 and Red Hat Enterprise Linux (RHEL) versions 9.6 and 8.10. This allows customers who have standardized on Ubuntu Linux to now run an InsightIQ 6.1 Scale deployment on a v24.04 host to monitor the latest OneFS versions.

Qualified on InsightIQ 6.0 InsightIQ 6.1
OS (IIQ Scale Deployment) RHEL 8.10, RHEL 9.4, and SLES 15 SP4 RHEL 8.10, RHEL 9.6, and SLES 15 SP4
PowerScale OneFS 9.4 to 9.11 OneFS 9.5 to 9.12
VMware ESXi ESXi v7.0U3 and ESXi v8.0U3 ESXi v8.0U3
VMware Workstation Workstation 17 Free Version Workstation 17 Free Version
Ubuntu – Ubuntu 24.04 Online deployment
OpenStack RHOSP 17 with RHEL 9.4 RHOSP 21 with RHEL 9.6

Similarly, in addition to deployment on VMware ESXi 8, the InsightIQ Simple version can also be installed for free on VMware Workstation 17, providing the ability to stand up InsightIQ in a non-production or lab environment for trial or demo purposes, without incurring a VMware licensing charge.

Additionally, the InsightIQ OVA template is now reduced in size to under 5GB, and with an installation time of less than 12 minutes.

Online Upgrade

The IIQ upgrade in 6.1 is a six step process:

First, the installer checks the current Insight IQ version, verifies there’s sufficient free disk space, and confirms that setup is ready. Next, IIQ is halted and dependencies met, followed by the installation of the 6.1 infrastructure and a migration of legacy InsightIQ configuration and historical report data to the new platform. The cleanup phase removes the old configuration files, etc, followed by the final phase which upgrades alerts and removes the lock, leaving InsightIQ 6.1 ready to roll.

Phase Details
Pre-check •       docker command

•        IIQ version check 6.0.1

•       Free disk space

•       IIQ services status

•       OS compatibility
Pre-upgrade •       EULA accepted

•       Extract the IIQ images

•       Stop IIQ

•       Create necessary directories
Upgrade •       Upgrade addons services

•       Upgrade IIQ services except alerts

•       Upgrade EULA

•       Status Check
Post-upgrade •       Update admin email

•       Update IIQ metadata
Cleanup •       Replace scripts

•       Remove old docker images

•       Remove upgrade and backup folders
Upgrade Alerts and Unlock •       Trigger alert upgrade

•       Clean lock file

The prerequisites for upgrading to InsightIQ 6.1 are either a Simple or Scale deployment with 6.0.1 installed, and with a minimum of 40GB free disk space.

The actual upgrade is performed by the ‘upgrade-iiq.sh’ script:

Specific steps in the upgrade process are as follows:

  • Download and uncompress the bundle
# tar xvf iiq-install-6.1.0.tar.gz
  • Enter InsightIQ folder and un-tar upgrade scripts
# cd InsightIQ
# tar xvf upgrade.tar.gz
  • Enter upgrade scripts folder
# cd upgrade/
  • Start upgrade. Note that the usage is same for both the Simple and Scale InsightIQ deployments.
# ./upgrade-iiq.sh -m <admin_email>

Upon successful upgrade completion, InsightIQ will be accessible via the primary node’s IP address.

Online Simple-to-Scale Migration

The Online Simple-to-Scale Migration feature enables seamless migration of data and functionalities from InsightIQ version 6.0.1 to version 6.1. This process is specifically designed to support migrations from InsightIQ 6.0.1 Simple (OVA) deployments to InsightIQ 6.1 Scale deployments.

Migration is supported only from InsightIQ version 6.0.1. To proceed, the following prerequisites must be met:

  • An InsightIQ 6.0.1 Simple deployment running IIQ 6.0.1.
  • An InsightIQ Scale deployment running IIQ 6.1.0 installed and the EULA accepted.

The ‘iiq_data_migration’ script can be run as follows to initiate a migration:

# cd /usr/share/storagemonitoring/online_migration
# bash iiq_data_migration.sh

Additionally, detailed logs are available at the following locations for monitoring and verifying the migration process:

Logfile Location
Metadata Migration Log /usr/share/storagemonitoring/logs/online_migration/insightiq_online_migration.log
Cluster Data Migration Log /usr/share/storagemonitoring/logs/clustermanagement/insightiq_cluster_migration.log

 Self-service Admin Password Reset

InsightIQ 6.1 introduces a streamlined self-service password reset feature for administrators. This secure process allows admins to reset their own passwords without IT intervention.

Key features include one-time password (OTP) verification, ensuring only authorized users can reset passwords. Timeout enforcement means OTPs expire after 5 minutes for added security, and accounts are locked after five failed attempts to prevent brute-force attacks.

Note that SMTP must be configured in order to receive OTPs via email.

Flexible Alerting

InsightIQ 6.1 enhances alerting capabilities with 16 total KPIs/metrics, including 9 new ones. Key improvements include:

  • Greater granularity (beyond cluster-level alerts)
  • Support for sub-filters and breakouts
  • Multiple operators and unit-based thresholding
  • Aggregator and extended duration support

Several metrics have been transformed and/or added in version 6.1. For example:

IIQ 6.1 Metric IIQ.6.0 Metric
Active Clients ·         Active Clients NFS

·         Active Clients SMB1

·         Active Clients SMB2
Average Disk Hardware Latency
Average Disk Operation Size
Average Pending Disk Operation Count ·         Pending Disk Operation Count
Capacity ·         Drive Capacity

·         Cluster Capacity

·         Node Capacity

·         Nodepool Capacity
Connected Clients ·         Connected Clients NFS

·         Connected Clients SMB
CPU Usage ·         CPU Usage
Disk Activity
Disk Operations Rate ·         Pending Disk Operation Count
Disk Throughput Rate
External Network Errors Rate
External Network Packets Rate
External Network Throughput Rate ·         Network Throughput Equivalency
File System Throughput Rate
Protocol Operations Average Latency ·         Protocol Latency NFS

·         Protocol Latency SMB

Also, clusters can now be directly associated with alert rules:

The generated alerts page sees the addition of a new ‘Metric’ field:

For example, an alert can now be generated at the nodepool level for the metric ‘External Network Throughput Rate’:

IIQ 6.1 also includes an updated email format, as follows:

Alert Migration

The alerting system has transitioned from predefined alerting to flexible alerting. During this migration, all alert policies, associated rules, resources, notification rules, and generated alerts are automatically migrated—no additional steps are required.

Key differences include:

IIQ 6.1 Flexible Alerting IIQ 6.0 Predefined Alerting
·         Each alert rule is associated with only one cluster (1:1 mapping). ·         Alert rules and resources are tightly coupled with alert policies.
·         A policy can still have multiple rules, but resources are now linked directly to rules, not policies.

·         This results in N × M combinations of alert rules and clusters (N = resources, M = rules).

 ·         A single policy can be linked to multiple rules and resources.

For example, imagine the following scenario:

  • Pre-upgrade (Predefined Alerting):

An IIQ 6.0 1 instance has a policy (Policy1), which is associated with two rules (CPU Rule & Capacity Rule) and 4 clusters (Cluster1-4).

  • Post-Upgrade (Flexible Alerting):

Since only one resource can be associated with one alert rule, a separate alert rule will be created for each cluster. So, after upgrading to IIQ 6.1, Policy1 will now have four individual cluster CPU Alert rules and four individual cluster Capacity Alert rules:

If an IIQ 6.1 upgrade happens to fail due to alert migration, a backup of the predefined alerting database is automatically created. To retry the migration, run:

# bash /usr/share/storagemonotoring/scripts/retrigger_alerts_upgrade.sh

Plus, for additional context and troubleshooting, the alert migration logs can be found at:

 /usr/share/storagemonitoring/logs/alerts/alerts_migration.log

Durable Data Collection

Data collection and processing in IIQ 6.x provides both performance and fault tolerance, with the following decoupled architecture:

Component Role
Data Processor Responsible for processing and storing the data in TimescaleDB for display by Reporting service.
Temporary Datastore Stores historical statistics fetched from PowerScale cluster, in-between collection and processing.
Message Broker Facilitates inter-service communication. With the separation of data collection and data processing, this allows both services to signal to each other when their respective roles come up.
Timescale DB New database storage for the time-series data. Designed for optimized handling of historical statistics.

InsightIQ TimescaleDB database stores long-term historical data via the following retention strategy:

Telemetry data is summarized and stored in the following cascading levels, each with a different data retention period:

Level Sample Length Data Retention Period
Raw table Varies by metric type. Raw data sample lengths range from 30s to 5m. 24 hours
5m summary 5 minutes 7 days
15m summary 15 minutes 4 weeks
3h summary 3 hours Infinite

Note that the actual raw sample length may vary by graph/data type – from 30 seconds for CPU % Usage data up to 5 minutes for cluster capacity metrics.

Meanwhile, the new InsightIQ v6.1 code is available for download on the Dell Support site, allowing both the installation of and upgrade to this new release.