OneFS Multi-party Authorization, or MPA, is an administrative approval mechanism that requires at least one additional trusted party to sign off on a requested change, for certain privileged actions within a PowerScale cluster. As such, MPA helps to reduce the likelihood of data loss or system configuration damage from critical actions, by mitigating the risk of accidental or malicious execution of consequential operations.
MPA can be managed via the OneFS WebUI, CLI, or platform API, and the WebUI portal can be found by navigating to Access > Multi-Party Authorization:
Similarly, the CLI uses ‘isi mpa’ command-set, with the following high-level options:
# isi mpa -h
Description:
Manage Multi-Party Authorization.
Usage:
isi mpa {<action> | <subcommand>}
[--timeout <integer>]
[{--help | -h}]
Actions:
complete-registration Complete Multi-Party Authorization registration.
initiate-registration Initiate Multi-Party Authorization registration.
Subcommands:
approvers Manage Multi-Party Authorization approvers.
ca-certificates Manage CA-Certificates for Multi-Party
Authorization.
requests Manage Multi-Party Authorization requests.
settings Manage Multi-Party Authorization settings.
Options:
Display Options:
--timeout <integer>
Number of seconds for a command timeout (specified as 'isi --timeout NNN <command>').
--help | -h
Display help for this command.
See 'isi mpa <subcommand> --help' for more information on a specific subcommand.
Within the platform API (pAPI), the MPA endpoints all reside under http://<cluster_IP>:8080/platform/23/mpa/. Their hierarchy and syntax is as follows:
/23/mpa/approval/<ID>
/23/mpa/approvers
/23/mpa/approvers/<ID>
/23/mpa/complete-registration
/23/mpa/initiate-registration
/23/mpa/requests
/23/mpa/requests/<ID>
/23/mpa/settings/config/request-lifecycle
/23/mpa/settings/global
/23/mpa/settings/privilege-action/metadata
/23/mpa/signed-approval/<ID>
/23/mpa/trust-anchors
An MPA pAPI endpoint can be queried as follows:
https://<cluster_IP>:8080/platform/23/mpa/settings/global
{
"last_update_time" : -1,
"mpa_enabled" : true
}
Invalid MPA calls will return a fairly descriptive error message and code. For example, the following ‘AEC_BAD_REQUEST’ error and 400 code resulting from a request query of a nonexistent service and action:
# curl -u root:a -k 'https://<cluster_IP>:8080/platform/mpa/requests' --data '{"service":"TestService", "action":"TestAction"}'
{
"errors" :
[
{
"code" : "AEC_BAD_REQUEST",
"message" : "Invalid privileged action TestAction for service TestService"
}
]
}
Additionally, the API of choice can be ‘described’ as follows:
https://<cluster_IP>:8080/platform/23/mpa/approvers?describe
Resource URL: /platform/23/mpa/approvers
Overview: List MPA approvers
Methods: GET
********************************************************************************
Method GET: List MPA approvers
URL: GET /platform/23/mpa/approvers
Query arguments:
registration_status=<enum> Where <enum> is one of Registration_Initiated, Pending_Approval, Registered, Registration_Denied.
Registration status of a MPA approver.
GET response body schema:
{
"type": [
{
"type": "object",
"description": "A list of errors that may be returned.",
--<snip>--
When upgrading a cluster from an earlier release, the MPA framework’s privileged actions and approval system (PAAS) becomes available after committing the upgrade to OneFS 9.12. As part of the upgrade process, a pre-check is performed to ensure that no existing role named ‘ApprovalAdmin’ (case-insensitive) is present. If such a role exists, administrators will be prompted to rename it to avoid conflicts with the newly introduced system role. This is necessary because privileges associated with the new ApprovalAdmin role will not be applied to pre-existing roles of the same name.
Upon upgrading to OneFS 9.12, the feature upgrade flag (ERA2_MPA) is automatically set to ‘true’ when the upgrade is committed, thereby providing, but not enabling, MPA functionality. Additionally, all MPA-related APIs become accessible only after the upgrade is finalized.
It is important to note that MPA is not supported in Compliance Mode. If attempting to enable MPA on a compliance cluster, the following warning message is displayed:
# isi mpa settings global modify --enable=true
Cannot enable Multi-Party Authorization mode on clusters with compliance mode already enabled
Prior to initiating an upgrade to OneFS version 9.12, the system performs a validation check to ensure that no existing role named ApprovalAdmin—regardless of case sensitivity—is present. The Multi-Party Authorization (MPA) feature becomes available only after the upgrade has been successfully committed. MPA does not require any special licensing; however, for enhanced security, it is recommended to enable MPA in conjunction with Root Lockdown Mode (RLM), which is also introduced in OneFS 9.12. Activation of RLM requires that the compadmin user is enabled and that both the Hardening and SmartLock licenses are valid and active across the cluster.
Note that the ability to disable MPA is not supported in OneFS 9.12, so consider this before activating the MPA feature.
When troubleshooting MPA-related issues, particularly those involving Time-Based One-Time Passwords (TOTP), several factors should be considered. For example:
If a ‘TOTP is invalid’ error occurs during the approver registration process, verify that the TOTP code is being generated from the correct account in the third-party application. Time synchronization between the client and the cluster is critical, and any time skew can result in token validation failures. If an approver loses their TOTP token or secret key, they must reinitiate registration. When MPA is enabled, this re-registration becomes a privileged action and requires approval.
In a rare scenario where all the MPA approvers have lost access to their TOTP credentials and no one is available to approve privileged actions, OneFS does provide a last resort ‘break-glass’ mechanism to handle such cases. Where appropriate, Dell can issue a specific patch that allows the approval of a registration request for a new ApprovalAdmin. This patch must be executed by the root user and will internally run the ‘isi mpa request approve’ CLI command for the specific request ID. This bypasses the TOTP check and is limited to the ‘register_approval_admin’ action. If such a scenario is encountered, please contact Dell Support immediately.
An MPA approval request can be in one of five states: Approved, Cancelled, Completed, Pending, or Rejected.
These states, and their various expiry times, can be viewed with the following CLI command syntax:
# isi mpa settings request-lifecycle list
Approved
Description: expire in configured days for privilege action execution.
Expire_time: 7d
Cancelled
Description: remove from system in configured days.
Expire_time: 7d
Completed
Description: remove from system in configured days.
Expire_time: 30d
Pending
Description: expire in configured days can be updated by user or approver.
Expire_time: 7d
Rejected
Description: remove from system in configured days.
Expire_time: 30d
The MPA requests comprise the following attributes.
|
| Request Attribute |
Description |
| action |
Name of privileged action |
| action_payload |
Set of key/value pairs for privileged action payload |
| cluster_guid |
Cluster GUID that uniquely identifies PowerScale |
| created_by |
Specified who created the request |
| creation_time |
Creation time of the request. Unix epoch time format |
| id |
Unique ID of MPA request. |
| last_update_time |
Last update time of the request. Unix epoch time format |
| request_for |
User for whom request is created for. |
| resource_ids |
List of resources IDs requested for approval. |
| resource_type |
Type of resource requested for approval. |
| service |
Name of service or component in system that owns the privileged action |
| status |
Status of MPA request |
| system_created |
A privileged action approval request was auto created by the system or not. |
| zone_id |
Multi-tenant access zone ID of MPA request created. |
If a privileged action fails to execute despite an approved request, it is necessary to verify whether the approval has expired. For troubleshooting purposes, diagnostic logs related to MPA can be found in the PAPI logs (/var/log/isi_papi_d.log). A fair amount of detail and context is captured by default without the need to increase logging verbosity. For example, the following secure snaphshot delete request, which includes snap ID, warning message, MPA request ID, URI info, error codes, etc:
2025-09-21T21:45:09.734523+00:00 <3.6> GLaDOS-8(id8) isi_papi_d[46945]: STACK snapshot ID: 2, Delete Snapshot is a privileged action. A request paareq43d57a6172a16de8 to perform this action is pending approval. Check Multi-Party Authorization to view status of the request. from --- (---:0): isi_exception::isi_exception(int, char const*, __va_list_tag*) (OFFSET:134) api_exception::api_exception(api_error_code, char const*, ...) (OFFSET:146) snapshot_mpa_interface::initiate_mpa_request(mpa_lib::MpaMatchContext&, std::__1::basic_string<char, mpa_lib::MpaMatchContext&::char_traits<char>, mpa_lib::MpaMatchContext&::allocator<char> > const&) (OFFSET:281) snapshot_snapshot_handler::http_delete(request const&, response&) (OFFSET:2136) uri_handler::execute_http_method_internal(request&, response&) (OFFSET:253) uri_handler::execute_http_method(request&, response&, bool, bool, bool) (OFFSET:1894) uri_manager::execute_request(request&, response&, bool, bool, bool, bool, bool, bool) (OFFSET:1578) std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf(void) (OFFSET:7452) std::__1::basic_filebuf<char, std::__1::char_traits<char> >::basic_filebuf(void) (OFFSET:11172) typeinfo name for api_exception (OFFSET:16919) typeinfo name for api_exception (OFFSET:13861) ADDRESS (UNKNOWN:-1806659576)
The OneFS services which include MPA privileged actions can be queried with the following CLI syntax:
# isi mpa settings pa-metadata list
For example:
For these MPA-integrated services, such as S3, service-specific logs can also provide useful additional diagnostics information. These MPA privileged actions and their associated logfiles in OneFS 9.12 include:
| Service /
Component |
Privileged Action |
Logfile |
| Hardening |
apply_hardening
disable_hardening |
/var/log/hardening.log
/var/log/hardening_engine.log |
| MPA |
register_approval_admin
upload_trust_anchor |
/var/log/isi_papi_d.log |
| S3 |
reduce_immutable_bucket_retention
modify_server_access_logging_config |
/var/log/s3.log |
| Snapshot |
delete_snapshot
modify_snapshot
delete_snapshot_schedule
modify_snapshot_schedule |
/var/log/isi_snapshot_d.log |
Complementary to MPA, OneFS 9.12 also introduces Root Lockdown Mode (RLM), which disables and blocks access to a cluster’s root account, forcing the use of roles-based access controls (RBAC). Since the root has explicit permission to bypass security controls, locking it down can also help to prevent a single bad actor from performing destructive operations on the data or storage platform.
As a best practice, enabling Root Lockdown Mode alongside Multi-party Authorization (MPA) provides a more secure and resilient configuration, ensuring full benefits of the MPA framework. To activate RLM, the compadmin user must be enabled, and both Hardening and SmartLock licenses are required. MPA and RLM can be enabled independently, and once MPA is activated, enabling RLM is a privileged action which requires an approval. RLM will be covered in-depth in a near-future article.
For disaster recovery in environments where both RLM and MPA are enabled, two procedures are available for recovering access to a node with a lost root password. After first contacting Dell Support, the preferred method involves smart-failing the node, re-imaging it, and rejoining it to the cluster. If this is not feasible, an alternative approach is to boot the node from alternate media and manually edit the System file provider to restore root access. After recovery, the Configurable Hardening Engine report should be reviewed to ensure all RLM rules are applied. If not, follow the documented steps to re-enable RLM.
To ensure the one-time usage of TOTP tokens, OneFS stores both the token and its generation counter in the MPA database under the approver’s record. When a new TOTP validation request is received, the system checks whether the token matches the previously used one. If it does, an error is triggered. The counter used to generate the token must be higher than the last used counter, ensuring that even unused older tokens cannot be reused once a newer token has been validated.
For example, if a token is generated at 12:00:00 and submitted at 12:00:20, the MPA uses the current time window (12:00:00–12:00:30) to generate the counter and validate the token. Once validated, both the token and counter are stored. If a malicious actor attempts to reuse the token, validation will fail either because the token matches the last used one or because the counter is outdated.
