In addition to the core file protocols, PowerScale cluster also supports the ubiquitous AWS S3 protocol. As such, applications have multiple access options to the same underlying dataset, semantics, across both file and object.
Also, since OneFS objects and buckets are essentially files and directories within the /ifs filesystem, the same OneFS data services, such as Snapshots, SyncIQ, WORM, etc, are all seamlessly integrated. This makes it possible to run hybrid and cloud-native workloads, which use S3-compatible backend storage – for example cloud backup & archive software, modern apps, analytics flows, IoT workloads, etc. – and to run these on-prem, alongside and coexisting with traditional file-based workflows.
The recent OneFS 9.11 release further enhances the PowerScale S3 protocol implementation with two new features: The addition of conditional write support and API-based cluster status reporting.
First, the new S3 conditional write support prevents the overwriting of existing S3 objects with identical key names. It does this via pre-condition arguments to the S3 ‘PutObject’ and ‘CompleteMultipartUpload’ requests with the addition of an ‘If-None-Match’ HTTP header. As such, if the condition is not met, the S3 operation(s) fail. Note, however, that OneFS does not currently support the ‘If-Match’ HTTP header, which checks the Etag value. More information about S3 conditional write is provided in the following AWS documentation: https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html
The second new piece of S3 functionality in OneFS 9.11 is API-based cluster status reporting. Increasingly, next gen applications need a reliable method decide where to store their backups and large data blobs across a variety of storage technologies. As such, a consistent API format, including cluster health status reporting, is needed to answer general questions about any S3 endpoint that maybe be under consideration as a potential target – particularly for applications without access to the management network. So providing the cluster status API facilitates intelligent decision making, such as how best to balance load and capacity across multiple PowerScale clusters. Additionally, the cluster status data can also help with performance analysis, as well as diagnosing hardware issues. For example, if an endpoint-alert has had zero successful objects delivered to it in the last hour, this status object will be the first thing that gets queried to see if there is a visible issue, or if applications are ‘routing around’ by intentionally using other resources.
The API uses an S3 endpoint with the following URL format:
S3://cluster-status/s3_cluster_status_v1
This mimics the GET object operation in the S3 service and is predicated on a via virtual bucket and object. As such, HEAD requests on this virtual bucket and object are valid, as is a GET request on the virtual object to read the Cluster Status data. All other S3 calls to this virtual bucket and object are prohibited, and the 405 HTTP error code returned.
Applications and users can use the S3 SDK, or other S3-conversant utility such as ‘s5cmd’, to retrieve the cluster status object, which involves the three valid S3 requests mentioned above:
- HEAD bucket
- HEAD object
- GET object
Where the ‘get object’ request returns the cluster status details. For example, using the ‘s5cmd’ utility from a windows client:
C:\s5cmd_2.3.0> .\s5cmd.exe –endpoint-url=http:10.10.20.30:9020 cat s3://cluster-status/s3_cluster_status_v1
{
“15min_avg_read_bw_mbs” : “0.12”,
“15min_avg_write_bw_mbs” : “0.04”,
“capacity_status_age_date” : “2025/06/04T07:43:02”,
“health” : “all_nodes_operational”,
“health_percentage” : “100”,
“health_status_age_date” : “2025/06/04T07:43:02”,
“mgmt_name” : “10.10.20.30:8080”
“net_state” : “full”,
“net_state_age_date” : “2025/06/04T07:43:02”,
“net_state_calculation” : {
“available_percentage” : “99”,
“down_bw_mbs” : “0”,
“total_bw_mbs” : “3576”,
“used_bw_mbs” : "0.01”,
},
“total_capacity_tb” : :0.06”,
“total_capacity_tib” : :0.05”,
“total_free_space_tb” : :0.06”,
“total_free_space_tib” : :0.05”,
}
The response format is JSON, and authenticated S3 users can access these APIs and download the cluster status object. The table below includes details of each response field:
| Requested Field | Description |
| mgmt_name | Management interface name of this cluster. |
| total_capacity_tb | Cluster’s total “current” capacity in base 10 terabytes. |
| total_capacity_tib | Cluster’s total “current” capacity in base 2 terabytes(tebibytes). |
| total_free_space_tb | Cluster’s total “current” free space in base 10 terabytes. |
| total_free_space_tib | Cluster’s total “current” free space in base 2 terabytes(tebibytes). |
| capacity_status_age_date | Number of second between the time of issuance and the proper calculation of capacity status. |
| health | Calculated status based on per node health status: either all_nodes_operational or some_nodes_nonoperational or non_operational. |
| health_percentage | Vendor specific number from 0-100% where the vendor’s judgement should be used has to what level of the systems normal load it can take. |
| health_status_age_date | Number of second between the time of issuance and the proper calculation of health status. |
| 15min_avg_read_bw_mbs | Read bandwidth in use, measured in megabytes per second, averaged over a 15-minute period. |
| 15min_avg_write_bw_mbs | Write bandwidth in use, measured in megabytes per second, averaged over a 15-minute period. |
| net_state | Networking status to S3 clusters. Divided into “Full”, “Half”, “Critical”, and “Unknown” |
| net_state_age_date | Number of second between the time of issuance and the proper calculation of network status. |
These fields can be grouped into the following core categories:
| Category | Description |
| Capacity | Reports the total capacity and available capacity in both terabytes and tebibytes. |
| Health | Includes the cluster health, node health and network health. |
| Management | ‘Management name’ references the out-of-band management interface that admins can use to configure the cluster. |
| Networking | Network status takes both the interfaces up/down status and the read write bandwidth on each interface into consideration. |
| Performance | Includes the read and write bandwidth. |
Under the hood, The high-level cluster reporting API operational workflow can be categorized as follows:
When an S3 client sends a get cluster status request, the OneFS S3 service retrieves the data from isi_status_d and Flexnet services. As part of this transaction, the calculations are performed and the result returned to the S3 client in JSON format. To speed up the retrieve process, a memory cache retains the data with a configured expiry time.
Configuration-wise, the addition of the cluster status API in OneFS 9.11 introduces the following new gconfig parameters:
| Name | Default Value | Description |
| S3ClusterStatusBucketName | “cluster-status” | Name of the bucket used to access cluster status. |
| S3ClusterStatusCacheExpirationInSec | 300 | Expiration time in seconds for cluster status cache in memory. Once reached, the next request for cluster status will results in a new fetch of fresh data. |
| S3ClusterStatusEnabled | 0 | Boolean parameter controlling whether the feature is enabled or not.
0 = disabled; 1 = enabled |
| S3ClusterStatusObjectName | “s3_cluster_status_v1” | Name of the object used to access cluster status. |
These parameter values can be viewed or configured using the ‘isi_gconfig’ CLI utility. For example:
# isi_gconfig | grep S3Cluster registry.Services.lwio.Parameters.Drivers.s3.S3ClusterStatusBucketName (char*) = cluster-status registry.Services.lwio.Parameters.Drivers.s3.S3ClusterStatusCacheExpirationInSec (uint32) = 300 registry.Services.lwio.Parameters.Drivers.s3.S3ClusterStatusEnabled (uint32) = 0 registry.Services.lwio.Parameters.Drivers.s3.S3ClusterStatusObjectName (char*) = s3_cluster_status_v1
The following gconfig CLI command syntax can be used to activate this feature, which is disabled by default:
# isi_gconfig registry.Services.lwio.Parameters.Drivers.s3.S3ClusterStatusEnabled=1 # isi_gconfig | grep S3Cluster | grep -i enabled registry.Services.lwio.Parameters.Drivers.s3.S3ClusterStatusEnabled (uint32) = 1
Two new operations are added to the S3 service, namely ‘head S3 cluster status’ and ‘get S3 cluster status’. For the head bucket, it will always return 200. For head cluster status object, the following three fields are required:
- ‘content-length’, which is the length of the cluster status object
- ‘last modified date’ maps the date for getting the cluster status object
- an empty ‘etag’
Note that OneFS uses the MD5 hash of an empty string for the empty value.
The S3 cluster status API is available once OneFS 9.11 has been successfully installed and committed. and S3 service is enabled. During upgrade to 9.11 the ‘404 Not Found’ will be returned if the API endpoints are queried.
There are a couple of common cluster status API issues to be aware of. These include:
| Issue | Troubleshooting step(s) |
| The get cluster status API fails to get the cluster status and returns 404 | Check if the configuration: S3ClusterStatusEnabled has been set to 1, or if the S3ClusterStatusBucketName and S3ClusterStatusObjectName match the bucket name and object name requested in the API. |
| The get cluster status API fails to get the cluster status and returns 403 | Check if the access key is input correctly and if the user is an authenticated user |
| The get cluster status API frequently returns “unknown” value | Verify that the dependent services (ie. isi_status_d) is running |
Helpful log files for further investigating API issues such as the above include the S3 protocol log, Stats daemon log, and Flexnet service log. These can be found at the following locations on each node:
| Logfile | Location |
| S3 protocol log | /var/log/s3.log |
| Flexnet daemon log | /var/log/isi_flexnet_d.log |
| Stats daemon log | /var/log/isi_stats_d.log |
Additionally, the following CLI utilities can also be useful troubleshooting tools:
# isi_gconfig # isi services s3