Category: PowerScale & Isilon

In this article, we’ll take a look at one of the quieter but genuinely useful serviceability enhancements that arrives with the OneFS 9.14 release: REST API access for creating and updating syslog rules. Delivered under capability 5512, this feature brings the cluster’s logging configuration into the same programmatic, automation-friendly world that storage admins increasingly expect from the rest of the platform

Before digging into what’s new, it’s worth a brief recap of what syslog actually does. Syslog is the ubiquitous standard that devices and applications use to ship event logs to a central location. On a PowerScale node, the syslog daemon receives log messages from the operating system and its various services through its sockets, and then routes each message to the appropriate destination: a local log file, or a remote log server.

Those routing decisions are governed by rules. Each message is tagged with a facility – the subsystem it originated from, such as auth, mail, kern, or daemon – and a severity level, running from debug all the way up to emergency. A syslog rule is essentially a filter-plus-destination pairing: messages from a given facility, at a given severity or above, are sent to a particular destination. On OneFS, as on most BSD-derived systems, these rules live in the /etc/syslog.conf file, which the daemon reads and evaluates at runtime.

Centralized, well-routed logging matters for more than just day-to-day troubleshooting. It underpins security monitoring, forensic investigation, and the audit trails that many compliance regimes mandate – which is precisely why reliably forwarding the right messages to the right collector is something operations teams care about getting right. It’s a simple, battle-tested model; the friction has always been in how you manage it.

Historically, configuring syslog rules on a cluster involved editing system files directly. To add a rule that forwarded authentication events to a security information and event management (SIEM) collector, for example, you’d open syslog.conf, hand-craft the appropriate selector and action lines, save the file, and validate the result manually. That approach works, but it carries a familiar set of problems:

• Error-prone – hand-editing configuration files is exactly the kind of task where a stray tab, a typo in a facility name, or a malformed destination quietly breaks logging, often in ways you don’t notice until you go looking for logs that were never written.
• Operational overhead – every change is a manual, repeated sequence of steps, and every cluster has to be touched individually.
• Inconsistent setups – without a programmatic source of truth, it’s hard to guarantee that twenty clusters share the same logging policy, and configuration drift inevitably creeps in.
• Poor automation support – manual file edits don’t slot neatly into the Infrastructure-as-Code and DevOps pipelines that modern operations teams build everything else around.

For a single cluster managed by one administrator, none of this is fatal. At scale, it becomes a genuine source of toil and risk.

OneFS 9.14 addresses this by exposing syslog configuration through the OneFS Platform API (PAPI). Rather than editing files, you now create, update, and delete syslog rules – including rules that forward to remote servers – through a documented, versioned REST interface. The benefits are exactly what you’d expect from moving any configuration surface from files to an API:

• Programmatic configuration that can be scripted and version-controlled.
• Seamless integration into existing automation workflows, whether that’s a provisioning script, a configuration-management tool, or a CI/CD pipeline.
• Consistency and repeatability, so the same policy can be applied identically across every cluster in the estate.
• Easier maintenance, because the API enforces structure rather than relying on free-form text.

Under the covers, the flow is straightforward. A client (i.e. a script, automation tool, or a simple curl one-liner) issues a request to the PAPI endpoint.

The platform API daemon, isi_papi_d, validates the request and translates it into the corresponding change in the syslog configuration. OneFS’s Master Control Program (MCP) is then responsible for propagating that configuration consistently across the nodes of the cluster, and the syslog daemon, isi_syslogd, picks up the updated rules and begins routing messages accordingly.

The key thing to note is that the API operates on the canonical configuration. There’s no risk of the API view and the on-disk configuration diverging, because the API is the thing writing the configuration. And because MCP handles distribution, a rule you create once is applied cluster-wide rather than node-by-node – which is precisely the consistency guarantee that hand-editing files on individual nodes could never offer.

The syslog auto rule functionality introduces two new endpoint groups. The first, /syslog/servers, manages the definition of remote syslog destinations, while the second, /syslog/rules, manages the rules themselves:

GET    /platform/<ver>/syslog/servers      # list configured remote servers
POST   /platform/<ver>/syslog/servers      # define a new remote server
DELETE /platform/<ver>/syslog/servers/<id> # remove a remote server

GET    /platform/<ver>/syslog/rules        # list API-created rules
POST   /platform/<ver>/syslog/rules        # create a new rule
DELETE /platform/<ver>/syslog/rules/<id>   # remove a rule

The division of responsibility is clean: The remote collectors are registered once via /syslog/servers, and then referenced when creating forwarding rules via /syslog/rules. A rule can forward matching messages to one of two kinds of destination – a local log file (restricted, sensibly, to the /var/log directory), or a remote syslog server that has been previously defined through the servers endpoint.

That /var/log restriction is worth calling out. By constraining local destinations to /var/log, OneFS avoids a whole class of problems in which a misconfigured rule could write log data into arbitrary – and potentially sensitive or space-constrained – areas of the filesystem. It’s a small guardrail, but a sensible one.

A typical sequence to forward authentication logs to a central collector breaks down into two steps. First, the remote server is registered through /syslog/servers, giving OneFS the address – and, depending on your environment, the transport and port – of the collector you want to ship logs to. Then, create a rule through /syslog/rules that selects the facility and severity you care about (authentication messages at, say, informational level and above) and points them at the server you just registered.

Removing the configuration later is the mirror image: delete the rule, then delete the server definition once nothing else references it. Listing is a simple GET against either endpoint, which makes the API equally valuable for auditing – you can ask a cluster what syslog rules it actually has, programmatically, rather than parsing a configuration file and hoping you read it correctly.

Because every one of these operations is a REST call, the whole sequence can be wrapped in a script and applied uniformly across as many clusters as you like. That is the entire point of the capability: the same logging policy, defined once, applied everywhere, with no text editors and no drift.

When investigating or troubleshooting syslog issues, there are a handful of places to look. To confirm that the syslog daemon itself is healthy, first check its service status:

# isi service isi_syslogd status

If an API call isn’t producing the configuration you expect, the platform API daemon’s log is the first stop, since that’s where request handling and validation are recorded:

# less /var/log/isi_papi_d.log

And because MCP is responsible for distributing the configuration across the cluster, its logs are the place to look if a change appears to have been accepted but has not been applied consistently:

# less /var/log/isi_mcp

Between the three – the daemon’s health, the API request log, and the MCP distribution log – you can usually pinpoint whether an issue lies in the request, in its propagation, or in the service itself.

In summary, the automated syslog rule configuration capability in OneFS 9.14 is a modest-sounding change with an outsized practical impact for anyone running PowerScale at scale. By exposing syslog server and rule management through the Platform API, OneFS turns what used to be a manual, error-prone, file-editing chore into a clean, scriptable, repeatable operation. Remote forwarding to a SIEM or central log server becomes a couple of API calls; auditing a cluster’s logging policy becomes a single GET; and rolling the same configuration out across an entire fleet becomes a loop in a script. So, for organizations standardizing on Infrastructure-as-Code, it removes one more manual island from the management plane. Plus, for everyone else, it simply makes a tricky task quicker, safer, and more easily and accurately repeatable.

Modern web applications routinely load resources — images, videos, fonts, scripts — from origins other than the server hosting the page. Browsers enforce the ‘same-origin policy’ by default, which blocks these cross-origin requests unless the target server explicitly permits them. Cross-Origin Resource Sharing (CORS) is the W3C mechanism that provides those permissions, and as of OneFS 9.14, PowerScale S3 supports it natively on a per-bucket basis.

This capability matters for any organization running a web application that reads from or writes to a PowerScale S3 bucket directly from a browser. Without CORS configuration on the bucket, the browser silently blocks the request at the network layer — no error in the server logs, just a failed request on the client side. A CORS policy tells the browser which origins, HTTP methods, and headers are acceptable, enabling seamless integration between frontend applications and PowerScale object storage while preserving the bucket’s underlying access controls.

Web browsers enforce a same-origin policy that, by default, prevents a web application served from one origin (a combination of scheme, host, and port) from reading resources served from a different origin. CORS relaxes this policy in a controlled manner. By using a defined set of HTTP response headers, a resource declares which other origins, HTTP methods, and request headers are permitted to access it.

Before sending a cross-origin request that could have side effects — such as a PUT or DELETE — the browser sends an HTTP OPTIONS preflight request to the target server. The preflight carries:

• Origin — the domain making the request
• Access-Control-Request-Method — the HTTP method the browser intends to use
• Access-Control-Request-Headers — any non-standard request headers

The S3 service evaluates these headers against the bucket’s CORS configuration and returns the appropriate Access-Control-* response headers. If all three are permitted by a matching rule, the browser proceeds with the actual request. If any one is rejected, the browser aborts. For simple read-only requests (GET, HEAD) the browser skips the preflight and reads the CORS response headers from the actual response directly.

CORS support extends OneFS S3 to browser-based workloads that would otherwise be blocked by the same-origin policy. Consider, for example, a website which needs to load images and other assets from a PowerScale S3 bucket. Without a CORS configuration the browser blocks these requests. Once CORS is configured to allow the application’s origin, the assets are retrieved and displayed successfully. Similarly, workflows where users upload files directly to a PowerScale S3 bucket from a front-end application, with no back-end proxy server in the request path. Once the bucket permits the application’s origin, uploads succeed and the browser receives a valid response with no CORS errors.

OneFS 9.14 introduces three new S3 API operations for managing per-bucket CORS configuration, and extends all existing bucket and object APIs with CORS response headers.

The CORS configuration is attached to an individual bucket and defines one or more rules. Each rule declares the allowed origins, HTTP methods, request headers, response headers to expose, and how long a browser may cache a preflight result.

1.       PutBucketCors

PutBucketCors sets a CORS configuration on a bucket. If a configuration already exists it is replaced in full — there is no partial update. The request body is an XML document:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <CORSRule>
    <ID>allow-webapp</ID>
    <AllowedOrigin>https://app.example.com</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <MaxAgeSeconds>3600</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>

A bucket can have multiple CORSRule blocks, each scoped to a different origin or method set. MaxAgeSeconds controls how long the browser may cache the preflight response, reducing round trips for repeat requests. Table 1 describes the CORS rule elements.

Element	Description
AllowedOrigin	One or more origins from which cross-origin requests are allowed. A single wildcard (*) is permitted.
AllowedMethod	An HTTP method the origin is allowed to use. Valid values:GET,PUT,HEAD,POST,DELETE.
AllowedHeader	A request header allowed in a preflight request viaAccess-Control-Request-Headers. A trailing wildcard (*) is permitted.
ExposeHeader	A response header that the browser is allowed to expose to the client application.
ID	An optional unique identifier for the rule.
MaxAgeSeconds	The time in seconds that the browser may cache the preflight response for the requested resource.

Required permission: Bucket owner or FULL_CONTROL ACL on the
bucket. Using the AWS CLI against a PowerScale endpoint:

aws s3api put-bucket-cors \
  --bucket mybucket \
  --cors-configuration file://cors.json \
  --endpoint-url https://powerscale.example.com:9020

2.       GetBucketCors

GetBucketCors returns the current CORS configuration for a bucket:

GET /<bucket>?cors

The response returns the same XML structure set by PutBucketCors. If no CORS configuration exists, the API returns 404 NoSuchCorsConfiguration.

Required permission: Bucket owner or READ ACL.

aws s3api get-bucket-cors \
  --bucket mybucket \
  --endpoint-url https://powerscale.example.com:9020

3.       DeleteBucketCors

DeleteBucketCors removes the CORS configuration from a bucket entirely:

DELETE /<bucket>?cors

After deletion the bucket returns no CORS headers and browser cross-origin requests will fail. There is no versioning, so the configuration is gone immediately.

Required permission: Bucket owner or FULL_CONTROL ACL.

The other new S3 operation introduced in OneFS 9.14 is the HTTP OPTIONS method, which browsers send as a preflight check before a cross-origin request that could have side effects. OneFS evaluates the preflight entirely outside the normal S3 authentication path, with no credentials required:

OPTIONS /<bucket>/<object> HTTP/1.1
Origin: https://app.example.com                  # origin url, including scheme and port
Access-Control-Request-Method: PUT               # intended http method
Access-Control-Request-Headers: Content-Type     # non-standard request headers

OneFS compares the origin, method, and headers against the bucket’s CORS rules and responds with the appropriate ‘Access-Control-*’ headers. Authentication is bypassed for ‘OPTIONS’ by design, consistent with the S3 specification, since the browser sends the preflight before it has any context for signing the subsequent request.

In addition to the three new CORS operations, all existing bucket-level and object-level S3 APIs now return CORS response headers on 200 OK responses where the request matches a configured rule. Three bucket-level APIs (HeadBucket, PutBucket/CreateBucket, and DeleteBucket) are intentionally excluded, consistent with Amazon S3 behavior.

Response Header	Description
Access-Control-Allow-Origin	The origin permitted to access the resource, echoed from the matched rule, or a wildcard (*).
Access-Control-Allow-Methods	The HTTP methods permitted for the matched origin.
Access-Control-Allow-Headers	The request headers permitted in the actual cross-origin request.
Access-Control-Expose-Headers	The response headers the browser is allowed to expose to the client application (e.g.ETag).
Access-Control-Max-Age	How long in seconds the browser may cache the preflight response.
Access-Control-Allow-Credentials	Whether the response may be exposed when the request’s credentials mode is set.

CORS support requires no additional cluster-level configuration. The feature becomes active once the S3 service is enabled on the cluster:

# isi s3 settings global modify --service enabled

A CORS rule applies to a bucket only after a client sets a configuration with PutBucketCors. Until then, the bucket enforces the browser’s default same-origin behavior. On upgrade, the feature is applied after the upgrade is committed. After uninstallation the feature no longer functions.

When investigating and troubleshooting S3 CORS, the most common issues typically fall into three categories:

• CORS configuration management errors
• OPTIONS preflight errors
• Rule matching failures.

1. CORS Configuration Management Errors

These occur when calling PutBucketCors, GetBucketCors, or DeleteBucketCors directly, typically because the supplied XML is malformed, the caller lacks the required ACL, or no configuration has been set yet:

Status	Error	Cause	Resolution
400	Bad Request	Incorrect CORS configuration XML	Check the error message and fix the configuration issue
403	Forbidden	Caller lacks required bucket ACL	FULL_CONTROL for Put/Delete, READ for Get; must be bucket owner
404	NoSuchCorsConfiguration	No CORS config set on the bucket	CallPutBucketCorsfirst

2. OPTIONS Preflight Errors

These are returned when the browser’s preflight request is missing one of the two required headers that OneFS needs to evaluate the request against the bucket’s CORS rules:

Status	Error	Cause	Resolution
400	InvalidRequest	OPTIONS preflight missingOriginorAccess-Control-Request-Method	Both headers are required in every preflight request

3. Rule Matching Failures

These are the most frequent issues in practice. The bucket has a CORS configuration but the incoming request does not satisfy any rule, either because no configuration exists at all, or because the origin, method, or headers are not listed as permitted. CorsNotAllowed is the most common: AllowedOrigin values are matched exactly, so https://app.example.com and https://app.example.com:443 are treated as different origins. Wildcards (*) are supported in AllowedOrigin and AllowedHeader but not in AllowedMethod:

Status	Error	Cause	Resolution
403	CorsNotEnabled	No CORS configuration on the bucket	CallPutBucketCorsto define at least one rule
403	CorsNotAllowed	Origin, method, or headers not matched by any configured rule	Review the CORS rules; confirmAllowedOriginmatches exactly (scheme + host + port)

In summary, PowerScale S3 CORS support in OneFS 9.14 adds three new S3 APIs, PutBucketCors, GetBucketCors, and DeleteBucketCors, plus the OPTIONS preflight handler. CORS response headers are also returned upon the successful responses of all existing bucket and object operations where a matching rule exists.

Configuration is per-bucket, requires no additional licensing, and takes effect immediately after PutBucketCors is called. The implementation is compatible with standard AWS S3 SDK clients using the ‘–endpoint-url’ override, making it straightforward to integrate into existing web application stacks that already target Amazon S3.

The OneFS Health Check Framework (HCF) has long provided a proactive way to monitor cluster health and surface potential issues before they affect data availability or performance. Behind the scenes, the framework periodically checks Dell’s Connectivity Hub portal for updated health check definition packages and repair action patches that have been assigned to the cluster, and when newer versions are found, automatically downloads and installs them without requiring any administrator involvement. For enterprise PowerScale environments in particular, this helps reduces maintenance overhead across large or geographically distributed deployments.

That said, the convenience of automatic updates is not universally applicable. Customers operating in regulated industries, high-security environments, or organizations with formal change management processes have consistently asked for the ability to control when and whether software components on their clusters are updated automatically. For these environments, having a patch silently installed — even a health check definition — can conflict with internal approval workflows, audit requirements, or network policies that restrict outbound connections to external systems. OneFS 9.14 addresses this directly with the introduction of independent controls for automatic download of health check definitions and repair action patches, giving administrators the ability to enable or disable each subsystem’s auto-update behavior without disabling the Health Check Framework itself.

Before looking at the new controls, it helps to understand the existing auto-update flow.

When Dell releases an updated health check definition package or a new repair action patch, it is made available through the Connectivity Hub portal and can be assigned to specific clusters. Once assigned, the cluster’s auto-update scripts, ‘isi_healthcheck_update’ for health check definitions and ‘isi_repair_update’ for repair actions, detect the new version on their next scheduled execution and proceed to download and install the update.

These scripts are the enforcement point for the new configuration, via gconfig control parameters, that the update scripts consult before taking any action. If auto-update is disabled for a given component when the script runs, the script exits cleanly without downloading or installing anything, even if a newer version has been assigned through the portal. This behavior is consistent whether the scripts are triggered by a scheduled job or invoked manually, so there are no edge cases where a manually triggered update bypasses the administrator’s preference.

The new configuration controls follow the same layered architecture used throughout OneFS for platform settings. Both the webUI and the CLI leverage the underlying platform API, which route requests to the Health Check Auto Update Handler and the Repair Settings Handler. Each handler exposes a PUT API endpoint that writes the relevant configuration values into the gconfig tree under the healthcheck and repair subtrees respectively. The update scripts then read these values at runtime to determine whether to proceed with the download and install.

This architecture means there is no change to how assignments are managed through the Connectivity Hub, and no change to how the scripts themselves are scheduled. The only difference in behavior is gated entirely by the gconfig values that the new API surfaces expose.

The new HCF functionality introduces independent controls for two distinct subsystems:

Subsystem	Details
Health Check Definitions	The check packages that define what the HCF tests on the cluster. Auto-update for health check definitions is enabled by default, meaning clusters running OneFS 9.14 or later will continue to receive updated check packages automatically unless this setting is explicitly disabled.
Repair Actions	Automated remediation procedures that can be applied when a health check identifies a problem. The repair service itself is disabled by default in OneFS, and its auto-update setting is also disabled by default. Both must be explicitly configured before any repair-related automatic updates will occur.

The separation between these two controls is intentional, allowing a cluster admin who wishes the cluster to always run the latest health checks, while maintaining tighter control over automated remediation, to leave health check auto-update enabled and keep repair auto-update disabled. Alternatively, an environment that prefers to fully lock down all automatic software changes can disable both.

Within the OneFS 9.14 WebUI, the new settings appear under Cluster management > HealthCheck > Settings  as two separate checkboxes:

• Auto update for health checks— enabled by default
• Auto update for repair actions— disabled by default

The repair actions checkbox is inactive (grayed out) until the repair service has been separately enabled. Attempting to toggle the repair auto-update setting before enabling the service generates an error indicating that auto-update settings can only be modified when the repair service is active.

These configuration options are also available from the CLI. For example, to disable automatic download of health check definitions from its default enabled state:

# isi healthcheck settings modify --enabled false

And similarly to re-enable it again:

# isi healthcheck settings modify --enabled true

For repair actions, the configuration is a two-step process. First, the repair service must be activated since it is disabled by default:

# isi repair settings modify --repair-enable true

Once the repair service is running, its auto-update behavior can be configured:

# isi repair settings modify --auto-update-enable true

Attempting to set –auto-update-enable without first enabling the repair service will fail with an error. The default value for repair auto-update is false, so simply enabling the repair service does not by itself turn on automatic patch downloads — that remains an explicit opt-in.

To check the currently installed version of repair actions on the cluster:

# isi repair settings view

   Repair Behavior: auto

    Repair Enabled: Yes

Autoupdate Enabled: No

This is useful when verifying that a manually initiated update completed successfully, or when confirming the current baseline before scheduling a maintenance window for a controlled update.

Access control and security-wise, two separate RBAC privileges govern these settings:

Action	Privilege
Health check configuration	Requires the ISI_PRIV_SYS_SUPPORT privilege.
Repair configuration	Requires the ISI_PRIV_REPAIR privilege.

In addition to privileges, there are cluster-state security restrictions that take precedence over the gconfig settings. As such, the automatic download functionality is suppressed entirely when the cluster is operating in any of the following elevated security or transitory states:

• Compliance mode
• Hardening mode (STIG)
• Root Lockdown Mode (RLM)
• Upgrade pre-commit

In these modes, automatic downloads are blocked regardless of the configured auto-update values, ensuring that the cluster’s compliance posture cannot be undermined by a background update process. Note that the gconfig values will not change on clusters transitioning into or out of these modes — only the enforcement behavior changes based on cluster state — so the configured preferences will resume taking effect once the cluster exits the restricted state.

When investigating and troubleshooting auto-update behavior, typically the first diagnostic step is to inspect the relevant gconfig settings directly:

# isi_gconfig -t healthcheck

[root] {version:1}

healthcheck_autoupdate.enabled (bool) = true

healthcheck_autoupdate.compliance_update (bool) = false

healthcheck_autoupdate.alerts (bool) = false

healthcheck_autoupdate.max_download_package_time (int) = 600

healthcheck_autoupdate.max_install_package_time (int) = 3600

healthcheck_autoupdate.number_of_failed_upgrades (int) = 0

healthcheck_autoupdate.last_failed_upgrade_package (char*) =

healthcheck_autoupdate.download_directory (char*) = /ifs/data/auto_upgrade_healthcheck/downloads

This outputs all current health check configuration values, including the auto-update flag. Confirm that the value of the auto-update key reflects the intended setting.

For repair-related issues, a logical first step is to verify that the repair service itself is running:

# isi services –a isi_repair

Service 'isi_repair' is enabled.

If the repair service is not active, any attempt to modify auto-update settings will fail. The relevant gconfig settings can be inspected directly, as follows:

# isi_gconfig -t repair

[root] {version:1}

repair.repair_enabled (bool) = true

repair.repair_behavior (char*) = auto

repair.default_repair_time_limit (int) = 600

repair.autoupdate_enabled (bool) = false

repair.alerts (bool) = false

repair.max_download_package_time (int) = 600

repair.max_install_package_time (int) = 3600

repair.number_of_failed_upgrades (int) = 0

repair.last_upgrade_package (char*) =

Additionally, detailed logs from repair update operations are written to the /var/log/isi_repair.log, and provide granular detail on what occurred during a given update attempt. This info includes whether a remote file was found, whether the download completed, and whether installation succeeded.

Similarly, for health check updates, the ‘isi_healthcheck_update’ script writes to /var/log/ isi_healthcheck.log, the contents of which will indicate whether it exited early due to auto-update being disabled, or whether it proceeded to download and install a package.

OneFS 9.14 introduced HTTPS support for the Common AntiVirus Agent (CAVA), enhancing the existing scanning solution with secure communication between the PowerScale cluster and Dell Common Event Enabler (CEE) server and enabling operation in hardening mode.

CAVA a service that runs on the CEE server, often referred to as CEE‑CAVA. It receives file access requests from the CAVA client interface on OneFS (also known as OneFS‑CAVA) and attempts to access the specified file over SMB. During this process, the antivirus engine residing on the CEE server scans the file. If CEE‑CAVA can successfully access and scan the file without detecting any threats, it returns a response indicating that the file is not infected. Conversely, if a threat is identified or access conditions indicate an issue, it responds with a message indicating that the file is infected.

Prior to OneFS 9.14, communication between the OneFS‑CAVA agent and the CEE‑CAVA service was conducted over HTTP, which did not provide secure transport. As a result, CAVA was automatically disabled on clusters operating in hardening mode. However, customers have expressed a strong requirement to enable CAVA while maintaining compliance with hardening mode security standards.

CAVA HTTPS support requires CEE version 9.1.2 or later in conjunction with OneFS 9.14 or newer, while existing antivirus licensing continues to be sufficient.

Configuring and enabling CAVA with HTTPS entails the following high-level process:

Note that the CAVA over HTTPS capability only becomes available after the upgrade to OneFS 9.14 has been fully committed. Additionally, upgrading to OneFS 9.14 does not impact existing legacy scanning workflows that continue to use standard HTTP transport.

1. Setup CA certificates

From the OneFS CLI, import the CA certificate(s) using its full path on the cluster:​

# isi antivirus cava certificates ca import --name=ca_unit --certificate-path=/ifs/certs/ca.pem​

Imported certificate: 4c6d3f0d1c3128ed09e02a78a1d9e3104edbfeb9c327c5e735afb3b48cd5bdbc​

​Once done, verify that the CA certificate was imported successfully:

# isi antivirus cava certificates ca list​

ID Name Not Before Not After Status​

--------------------------------------------------------------------------------------------------------------------------​

4c6d3f0d1c3128ed09e02a78a1d9e3104edbfeb9c327c5e735afb3b48cd5bdbc ca_unit 2025-08-28T07:39:22 2125-08-04T07:39:22 valid​

--------------------------------------------------------------------------------------------------------------------------​

Total: 1​

# isi antivirus cava certificates ca view 4c6d3f0d1c3128ed09e02a78a1d9e3104edbfeb9c327c5e735afb3b48cd5bdbc​

ID: 4c6d3f0d1c3128ed09e02a78a1d9e3104edbfeb9c327c5e735afb3b48cd5bdbc​

Name: ca_unit​

Status: valid​

Not Before: 2025-08-28T07:39:22​

Not After: 2125-08-04T07:39:22​

Description:​

Fingerprints​

Type: SHA1​

Value: f1:0e:34:d0:0c:65:99:b8:86:58:f3:b3:36:21:9a:64:7a:e6:b5:03​

Type: SHA256​

Value: 4c:6d:3f:0d:1c:31:28:ed:09:e0:2a:78:a1:d9:e3:10:4e:db:fe:b9:c3:27:c5:e7:35:af:b3:b4:8c:d5:bd:bc​

Subject: C=IN, ST=DCU, L=Gotham, O=Wayne Enterprises, OU=Batman, CN=batmanCA, emailAddress=batmanCA@bat.man​

Issuer: C=IN, ST=DCU, L=Gotham, O=Wayne Enterprises, OU=Batman, CN=batmanCA, emailAddress=batmanCA@bat.man​

​Or from the WebUI under Data Protection > Antivirus > CAVA, where OneFS 9.14 introduces a new Certificates section which includes three tabs: Authority, Identity, and Settings:

Users with the ‘ISI_PRIV_ANTIVIRUS’ RBAC privilege can manage CA and identity certificates, including performing operations such as ‘import’, ‘edit’, ‘replace’, and ‘delete’.

Import the certificate authority (CA) under Data Protection > Antivirus > CAVA > Certificates > Authority:

2. Setup Identity Certificates

Next, import the identity certificate(s) as follows, specifying the full path to the cert and key files:​

# isi antivirus cava certificates identity import --name=onefs_cava --certificate-path=/ifs/certs/signed_id_cert.pem --certificate-key-path=/ifs/certs/id_cert.key --skip-certificate-passphrase​

Imported certificate: 70b6734399785ff5d512a789836bdf45e4ee35cf464e6da36d16cda5db6ef353​

Once complete, verify that the identity certificate was imported successfully:

# isi antivirus cava certificates identity list​

ID Name Not Before Not After Status​

---------------------------------------------------------------------------------------------------------------------------​

70b6734399785ff5d512a789836bdf45e4ee35cf464e6da36d16cda5db6ef353 onefs_cava 2026-01-21T13:51:12 2125-12-28T13:51:12 valid​

---------------------------------------------------------------------------------------------------------------------------​

Total: 1​

# isi antivirus cava certificates identity view 70b6734399785ff5d512a789836bdf45e4ee35cf464e6da36d16cda5db6ef353​

ID: 70b6734399785ff5d512a789836bdf45e4ee35cf464e6da36d16cda5db6ef353​

Name: onefs_cava​

Status: valid​

Not Before: 2026-01-21T13:51:12​

Not After: 2125-12-28T13:51:12​

Description:​

Fingerprints​

Type: SHA1​

Value: 79:68:ab:a2:5d:95:31:53:6c:90:a7:5c:99:d5:f7:95:a7:db:99:22​

Type: SHA256​

Value: 70:b6:73:43:99:78:5f:f5:d5:12:a7:89:83:6b:df:45:e4:ee:35:cf:46:4e:6d:a3:6d:16:cd:a5:db:6e:f3:53​

Subject: C=IN, ST=DCU, L=Gotham, O=Wayne, OU=Batman, CN=robinOnefs, emailAddress=robinOnefs@bat.man​

Issuer: C=IN, ST=DCU, L=Gotham, O=Wayne Enterprises, OU=Batman, CN=batmanCA, emailAddress=batmanCA@bat.man​

Or from the WebUI under Data Protection > Antivirus > CAVA > Certificates > Identity:

3. Add CAVA server

Next, create the CAVA server configuration on the cluster:​

# isi antivirus cava servers create CAVA1 https://10.10.20.50:12443/cee --enabled 1​

​Once done, verify that the CAVA server was added successfully:

# isi antivirus cava servers list​

Server Name Server URI Enabled Server Type​

------------------------------------------------------------------​

CAVA1 https://10.10.20.50:12443/cee Yes CEE/CAVA​

------------------------------------------------------------------​

Total: 1​

​

# isi antivirus cava servers view CAVA1​

Server Name: CAVA1​

Server URI: https://10.10.20.50:12443/cee​

Enabled: Yes​

Server Type: CEE/CAVA​

​Or from the WebUI under Data Protection > Antivirus > CAVA > Servers:

Note that, when adding the server, specify protocol as https:// and port number in the URI​.

If needed, a CAVA server can also be removed with the following CLI syntax:

# isi antivirus cava servers delete CAVA1​

4. Configure identity certificate ID and Enable TLS in CAVA global settings

The ‘–enforce-tls’ global configuration parameter is set to ‘false’ by default, permitting CAVA to use regular HTTP transport. By setting this flag to ‘true’, CAVA over HTTPS communication is enabled.

From the CLI:

# isi antivirus cava settings modify --enforce-tls=true --certificate-id=7ba9188a5f6bb120c8ea3e499b662b38319587804492cf444df02b33b1d4a86c​

​# isi antivirus cava settings view​

Service Enabled: Yes​

Scan Access Zones: System​

IP Pool: groupnet0.subnet0.cavapool​

Report Expiry: 8 weeks, 4 days​

Scan Timeout: 1 minute​

Cloudpool Scan Timeout: 1 minute​

Maximum Scan Size: 0.00kB​

Certificate ID: 7ba9188a5f6bb120c8ea3e499b662b38319587804492cf444df02b33b1d4a86c​

Enforce TLS: Yes​

Note that omitting the client certificate ID will result in the following error:

# isi antivirus cava settings modify --enforce-tls=true

TLS cannot be enabled as no client certificate ID is configured.

This can also be configured from the WebUI under Data protection > Antivirus > CAVA > Settings. Disabled by default, in order to activate this option, users must select an identity certificate, which is a prerequisite for enforcing TLS:

Note that attempts to activate TLS enforcement on a cluster without any identity certificates will generate the following pop-up warning:

This warning includes a link, which opens the identity certificate import and configuration portal.

Optionally add an OCSP server URI and set revocation mode:​

# isi antivirus cava certificates settings view​

OCSP Responder URI:​

OCSP Revocation Mode: None​

Strict Hostname Check: No​

# isi antivirus cava certificates settings modify --ocsp-responder-uri=http://10.1.100.20 --ocsp-revocation-mode=Strict​

5. CEE configuration

On the common event enabler (CEE) server side, there are a few prerequisites and registry configuration changes that are required to support CAVA over HTTPS.

a. First, the server needs to be running CEE version 9.2.1.0 or later.

b. The CEE port configuration needs to be configured to use HTTPS (tcp/12243) in the server’s Windows Registry.​ This can be achieved by running the following commands on the server:​

reg add HKEY_LOCAL_MACHINE\SOFTWARE\EMC\CEE\Configuration\Security\Https /v ServerEnabled /t REG_DWORD /d 1 /f​

c. Upload and configure the following CA and identity certificates on CEE server.

CA certificates: ​

• pem​
• key​

Identity Certificates​

• pem​
• key​

These certificates should be copied from the PowerScale cluster to a configured path on the CEE server, for example C:\CEE_CERTS\ as in the configuration example below:

​First, configure the paths for CEE identity certificate and key​ in the CEE server’s registry:

reg add HKEY_LOCAL_MACHINE\SOFTWARE\EMC\CEE\Configuration\Security\Https /v PrivateKey /t REG_SZ /d "C:\CEE_CERTS\cee_id_cert.key" /f​

reg add HKEY_LOCAL_MACHINE\SOFTWARE\EMC\CEE\Configuration\Security\Https /v Certificate /t REG_SZ /d "C:\CEE_CERTS\cee_signed_id_cert.pem" /f​

​Next, install the CA certificate into the Windows certificate store:​

> certutil -addstore "Root" "C:\CEE_CERTS\ca.pem"

d. Finally, confirm the CEE registry settings have been applied successfully from the Windows registry editor (regedit):

 

6. Verify CAVA status.

Once all the configuration steps above are completed, check the CAVA status​ and confirm ‘Good Heartbeats’ are being reported:

# isi antivirus cava status​

System Status: RUNNING​

Fault Message: -​

CEE Version: 9.2.0.0​

DTD Version: 2.5.3​

AV Vendor: MS Forefront​

Last Signature Update: Fri Feb 13 02:10:19 2026​


# isi antivirus cava status –-servers

Server Name   Server State Good Heartbeats     Heartbeat RTT Scan RTT     Scan Requests Connections

-------------------------------------------------------------------------------------

TLS           Active        235                  14.0ms        0.0ms         0             6

--------------------------------------------------------------------------------------


# /usr/likewise/bin/lw-av active-servers​

Active Anti-virus Servers:​

Number of servers: 1​

Server Name: CAVA1​

Server Type: CAVA​

Server Enabled: Yes​

Server State: Active​

Server URI: https://cava.lab.com:12443/cee​

Good Heartbeats: 235

Heartbeat Round-Trip-Time (ms): 14​

Number of Scan Requests: 0​

Average Scan Round-Trip-Time (ms): 0​

Target Connection Count: 6​

Connection Count from Node: 6

If needed, CAVA HTTPS enforcement can also be easily disabled from the CLI via the following global settings configuration change: ​

​# isi antivirus cava settings modify --enforce-tls=false​

In addition to WebUI and CLI configuration options, OneFS 9.14 also sees the introduction and modification of the following CAVA platform API (pAPI) endpoints related to HTTPS support:

Function	API Endpoint	Status
CA certificates	/25/avscan/cava/certificates/ca​ /25/avscan/cava/certificates/ca/<ID>​	New
Identity certificates	/25/avscan/cava/certificates/identity​ /25/avscan/cava/certificates/identity/<ID>​	New
Certificate settings	/25/avscan/cava/certificates/settings​	New
CAVA global settings	/25/avscan/cava/settings​	Modified
CAVA server management	/25/avscan/servers	Modified

Another feature introduced in the OneFS 9.14 release is CELOG Bulk Event Resolution. But before we get into the details, first, a quick refresher. The OneFS Cluster Event Log (or CELOG) provides a single source for the logging of events that occur on a PowerScale cluster. Events are used to communicate a picture of cluster health for various components. CELOG provides a single point from which notifications about the events are generated, including sending alert emails and SNMP traps.

Cluster events can be easily viewed from the WebUI by browsing to Cluster Management > Events and Alerts > Events group history. For example:

Or from the CLI, using the ‘isi event events view’ syntax:

# isi event events view 2.370158

           ID: 2.370158

Eventgroup ID: 271428

   Event Type: 600010001

      Message: The snapshot daemon failed to create snapshot 'Hourly - prod' in schedule 'Hourly @ Every Day': error: Name collision

        Devid: 2

          Lnn: 2

         Time: 2026-05-08T17:01:33

     Severity: warning

        Value: 0.0

In the above instance, CELOG communicates on behalf of SnapshotIQ that it’s failed to create a scheduled hourly snapshot because of an issue with the naming convention.

At a high level, processes that monitor conditions on the cluster or log important events during the course of their operation communicate directly with the CELOG system. CELOG receives event messages from other processes via a well-defined API.

A CELOG event often contains the following elements:

Element	Definition
Event	Events are generated by the system and may be communicated in various ways (email, snmp traps, etc), depending upon the configuration.
Specifier	Specifiers are strings containing extra information, which can be used to coalesce events and construct meaningful, readable messages.
Attachment	Extra chunks of information, such as parts of log files or sysctl output, added to email notifications to provide additional context about an event.

For example, in SnapshotIQ event above, we can see the event text contains a specifier and attachment that has been mostly derived from the corresponding syslog message:

# grep "Hourly - prod" /var/log/messages* | grep "2026-05-08T17:01:33"

2026-05-08T17:01:33-04:00 <3.3> a200-2 isi_snapshot_d[5631]: create_schedule_snapshot: snapshot schedule (Hourly @ Every Day) pattern created a snapshot name collision (Hourly - prod); scheduled create failed.

CELOG is a large, complex system, which can be envisioned as a large pipeline. It gathers events and statistics info on one end from isi_stats_d and isi_celog_monitor, plus directly other applications such as SmartQuotas, SyncIQ, etc. These events are passed from one functional block to another, with a database at the end of the pipe. Along the way, attachments may be generated, notifications sent, and events passed to a coalescer.

On the front end, there are two dispatchers, which pass communication from the UNIX socket and network to their corresponding handlers. As events are processed, they pass through a series of coalescers. At any point they may be intercepted by the appropriate coalescer, which creates a coalescing event and which will accept other related events.

As events drop out the bottom of the coalescer stack, they’re deposited in add, modify and delete queues in the backend database infrastructure. The coalescer thread then moves onto pushing things into the local database, forwarding them along to the master coalescer, and queueing events to have notifications sent and/or attachments generated.

The processes of safely storing events, analyzing them, deciding on what alerts to send and sending them is separated into four separate modules within the pipeline:

The following table provides a description of each of these CELOG modules:

Component	Definition
Capture	The first stage in the processing pipeline, Event Capture is responsible for reading event occurrences from the kernel queue, storing them safely on persistent local storage, generating attachments, and queueing them by priority for analysis.
Analysis	Extra chunks of information (log file extracts, sysctl output, etc) are added to alert notifications to provide additional context about an event.
Reporter	The Reporter is the third stage in the processing pipeline, and runs on only one node in the cluster. It periodically queries Event Analysis for changes and generates alert requests for any relevant conditions.
Alerter	The Alerter is the final stage in the processing pipeline, responsible for actually delivering the alerts requested by the reporter. There is a single sender for each enabled channel on the cluster.

CELOG local and backend database redundancy ensures reliable event storage and guards against bottlenecks.

By default, OneFS provides the following event group categories, each of which contain a variety of conditions, or ‘event group causes’, which will trigger an event if their conditions are met:

Event Group Category	Event Series Number
System disk events	1000*****
Node status events	2000*****
Reboot events	3000*****
Software events	4000*****
Quota events	5000*****
Snapshot events	6000*****
Windows networking events	7000*****
Filesystem events	8000*****
Hardware events	9000*****
CloudPools events	11000*****

Prior to OneFS 9.14 and the introduction of CELOG bulk event resolution, cluster events were handled individually, resulting in excessive parallel API calls, frequent timeouts, repeated database commits, and increased lock contention, which constrained system scalability under high event volumes and degraded the user experience through delays, failures, and additional manual verification. Bulk Event Resolution addresses these limitations by processing multiple events within a single atomic transaction in which all changes either succeed or fail together, ensuring data integrity while minimizing redundant API activity and providing consistent rollback on failure. This capability significantly reduces database lock contention, improves stability and throughput during high event loads, and accelerates the completion of administrative operations. The feature is fully integrated across the WebUI, PAPI, and CLI, enabling consistent interactive, programmatic, and script‑driven workflows, and delivering a more reliable and efficient operational experience with reduced need for manual validation.

Under the hood, CELOG has the following high level architecture:

The CELOG bulk resolve workflow itself operates as follows:

Both the WebUI and CLI route bulk event requests through the isi_papi_d service, where the event occurrences handler orchestrates the overall operation. The handler begins by authenticating the request and validating the input, then verifies all provided event group IDs and filters out any that are invalid. Only confirmed IDs are allowed to proceed, and these are deterministically grouped and executed together within a single transaction. The workflow is initiated through a request to the bulk endpoint, and the remaining valid events are processed as part of one automated transaction. If all steps complete successfully, the API returns a success response; if any step fails, the transaction is rolled back and an appropriate error is returned. No additional configuration is required, as bulk resolution leverages the existing PUT API and follows the same authentication and permission model. When multiple event IDs are supplied as an array, the backend automatically processes them in bulk mode, without requiring feature flags, database changes, or service configuration updates.

Bulk event actions can be managed from the OneFS WebUI under ‘Cluster management > Events and alerts > Event group history’:

Multiple event groups can be selected and resolved or ignored as a single collective action:

Or from the OneFS ClI using the following syntax:

# isi event groups bulk -–resolved true -–eventgroup_ids=<x,x,x,...>

After the action is confirmed, the WebUI submits the bulk request and processes it asynchronously in the background:

Upon successful completion of the bulk resolution operation, the WebUI displays a success banner to confirm the result:

If the request includes invalid event IDs, those IDs are filtered out during validation, and the response clearly identifies which events were successfully resolved and which were skipped. For example, from the CLI:

# isi event groups bulk -–resolved true -–eventgroup_ids=55,56,250

Resolved event-groups: 55,56

Skipped event-groups: 250

#

By consolidating processing into a single atomic operation, redundant requests are eliminated, performance is improved, and consistent, reliable outcomes are maintained even under high‑load conditions. This approach delivers a faster experience for cluster administrators, increases overall system resilience, and simplifies end‑to‑end automation.

If a bulk request fails, begin by confirming that the payload includes a valid array of event IDs and that the appropriate permissions are in place. Next, review the API log messages at /var/log/isi_papi_d.log. Because execution is atomic for the validated set, any failure during processing triggers a rollback of the entire transaction, with the API response and logs providing clear indicators of the underlying cause.

Introduced in OneFS 9.14, Durable Handles are a feature of the SMB2 and SMB3 protocols in which, when a client opens a file, it receives an opaque file handle that can be marked as durable. A durable handle allows the open file state to survive a temporary client disconnect, such as a brief network glitch, transient cluster interruption, wi‑fi connectivity drop, or client sleep event, enabling the SMB client to reconnect and reclaim the same handle within a server-defined grace period. From the application’s perspective, the file remains open and normal I/O continues without errors or forced reopens. Durable handles are requested by the client at open time using specific CREATE request contexts, such as SMB2_CREATE_DURABLE_HANDLE_REQUEST and the corresponding reconnect variants.

Durable file handle support in OneFS 9.14 and later provides the following attributes and benefits:

Attribute	Details
Availability	Supported with SmartConnect static IP pools in OneFS 9.14.​
Client	Supports v2 Durable Handles (SMB 3+ dialects)​.
Context	Client can reconnect to the same file without losing its context​.
Cost	Lightweight, avoiding the performance overhead of full Continuous Availability​.
Persistence	Allows the cluster to keep the SMB file handle alive briefly​.

Under the hood, OneFS durable handles employ the following fundamental architecture:

At the protocol level, a durable handle is established when the client sends an SMB CREATE request that includes a durable handle create context. If the server and share configuration permit durable handles, the cluster’s SMB server marks the open accordingly and returns a persistent handle identifier along with a reconnect token, such as a create GUID, which can later be used to reclaim the handle. During normal operation, the client performs READ, WRITE, and locking operations using that handle over its SMB session.

If the client disconnects unexpectedly and the underlying TCP connection and SMB session are lost, the cluster retains the durable handle’s state, including open and lock information, for a defined timeout period rather than closing it immediately. When the client reconnects, it reestablishes an SMB session and tree connection and issues a new CREATE request with a durable handle reconnect context that includes the original token:

If the cluster still holds the handle and the reconnect request matches, it rebinds the durable handle to the new session and I/O seamlessly resumes. If the client does not reconnect before the timeout expires, the cluster closes the open, releases any associated locks, and the handle can no longer be reclaimed.

As such, durable handles are intended to protect against short client-side outages and assume that the SMB server instance remains available. They do not survive a full-on SMB server and/or node failure unless additional high-availability mechanisms such are in place.

One such mechanism is SMB Continuous Availability (CA), a related and complimentary feature of the SMB3 protocol, which extends this model to provide high availability for clustered SMB file servers by allowing open file state and I/O to survive planned and unplanned server or node failovers.

Focus	SMB Durable Handles	SMB Continuous Availability
What	Best-effort resilience to transient client connectivity issues while assuming the server remains running.	Explicitly designed to tolerate server or node failures in a clustered environment.
How	Stores open state locally for a limited time and is negotiated per file open.	Requires share-level configuration and supporting cluster infrastructure to persist state across nodes.
Where	General file access scenarios where brief network disruptions are expected, such as user desktops or laptops.	CA shares are intended for critical workloads that demand uninterrupted access through failover events.
Version	OneFS 9.14 onwards.	OneFS 8.0 onwards.
Type	Lighter weight, without the write-stability requirements of CA	Uses persistent handles.
Performance	Much lower performance impact.	Higher performance cost due to stable write requirement.
Realm	Granted only when client connects through a static IP pool in OneFS 9.14.	Works with both dynamic and static IP pools.
Status	Enabled by default.	Disabled by default.

PowerScale has supported Continuous Availability since OneFS 8.0, and it is enabled at the share level by marking a share as continuously available, relying on persistent handle and lease state that is stored or replicated in a highly available, cluster-consistent manner. This allows the SMB server resource to move to another node during a failover while clients transparently reconnect and continue I/O without application disruption. CA combines persistent or durable v2 durable handles with clustering and witness mechanisms to coordinate reconnection and ensure strict data consistency semantics across nodes.

In practice, durable handles and SMB3 continuous availability differ in scope and guarantees. Durable handles provide best-effort resilience to transient client connectivity issues while assuming the server remains running, whereas CA is explicitly designed to tolerate server or node failures in a clustered environment. Durable handles store open state locally for a limited time and are negotiated per file open, while CA requires share-level configuration and supporting cluster infrastructure to persist state across nodes. As a result, durable handles are commonly used for general file access scenarios where brief network disruptions are expected, such as user desktops or laptops, while CA shares are intended for critical workloads like Hyper‑V or SQL Server over SMB that demand uninterrupted access through failover events.

On a PowerScale cluster running OneFS 9.14 or later, durable handles are supported as part of standard SMB2 and SMB3 operation to help clients recover from short connectivity disruptions. SMB3 continuous availability is provided through CA-enabled SMB shares, where OneFS ensures that file, handle, and share state are protected across service or node failovers within the cluster, allowing appropriately capable SMB3 clients to resume I/O transparently. As such, all CA shares make use of durable or persistent handle semantics internally, but durable handles alone do not imply continuous availability; CA represents a share-level, cluster-integrated high-availability capability, whereas durable handles are a file open resiliency mechanism.

Enabled by default in OneFS 9.14, durable file handles are only granted to clients that request them via an SMB session established from a static-IP SmartConnect network pool. Note that durable file handles on dynamic pools will be supported in a future OneFS release.

Durable handles can be configured on a per-share basis from the CLI, WebUI and platform API, as follows:

During creation:

# isi smb shares create --durable-handle-enabled <true | false>

Or from the OneFS WebUI under Protocols > Windows sharing (SMB) > SMB shares > Create a SMB share:

Durable handles support can modified on an existing share:

# isi smb shares modify --durable-handle-enabled <true | false>

A timeout can be configured with the ‘–ca-timeout’ flag, and the default duration is 120 seconds:

# isi smb shares create --durable-handle-enabled true --ca-timeout

As noted previously, SMB durable handles are enabled by default in OneFS 9.14:

# isi smb settings shares view | grep -i dura

Durable Handle Enabled: Yes

They can also be easily disabled globally as follows:

# isi smb settings shares modify --durable-handle-enabled 0

# isi smb settings shares view | grep -i dura

Durable Handle Enabled: No

Or from the WebUI under Protocols > Windows sharing (SMB) > Default share settings:

Additionally, Continuous Availability can also now be enabled or disabled on an existing share as follows:

# isi smb shares modify --continuously-available <true | false>

When enabling CA on a share, the following confirmation popup is displayed, advising of the potential write performance implications when activating Continuous Availability:

Note that, due to the IP allocation cache and timer of entry, a change in IP allocation method from dynamic to static or vice versa can potentially result in up to a five minute delay before durable handles configuration changes (enable or disable) are enacted.​

If and when it comes to investigating and troubleshooting durable handles, the /var/log/lwiod.log is a good place to start. ​Beyond that, network packet captures can also be extremely helpful at understanding and verifying the SMB sessions at a protocol request level. When examining a pcap of an SMB session with a network sniffing tool (e.g. Wireshark), the presence of the ‘SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2’ request with a ‘Persistent Handle’ flag of value zero (“0”) indicates that this is a Durable Handle request:

Alternatively, if the persistent handle flag contains a value of one (“1”), this indicates that the request is for a persistent handle, with the likelihood that SMB CA is involved.

As we saw in the previous article, OneFS 9.14 adds S3 Lifecycle Management, which allows administrators to define policies that automate object management within PowerScale S3 buckets. These policies enable the automatic deletion of objects based on criteria such as age, size, or key prefix and are applied uniformly to both existing and newly created objects in a bucket. Lifecycle processing is handled by the OneFS Job Engine, which runs daily to evaluate configured rules and generates per‑bucket tasks that traverse bucket directories and remove objects that meet the defined conditions.

To support this new functionality, the S3 API support in OneFS 9.14 and later now includes the following endpoints:

API Endpoint	Description
PutBucketLifecycleConfiguration	Sets the lifecycle configuration for the bucket and replaces any existing one. User must be the bucket owner to create the lifecycle configuration.
GetBucketLifecycleConfiguration	Returns the current lifecycle configuration for the bucket. User must be the bucket owner to get the lifecycle configuration. Will return a ‘NoSuchLifecycleConfiguration’ error if a configuration is not found.​
DeleteBucketLifecycle	Deletes the lifecycle configuration for the bucket. User must be the bucket owner to delete the lifecycle configuration.
AbortIncompleteMultipartUpload​	​
·         DaysAfterInitiation​	Number of days after the system aborts an incomplete MPU​.

Plus, the following S3 endpoints are also updated in 9.14 and require the following read and write permissions:

S3 Endpoint	Read Permission	Write Permission
CompleteMultiPartUpload	x	x
CopyObject	x	x
GetObject	x
HeadObject	x
PutObject	x	x

In this second article in the series, we’ll walk through a simple example demonstrating how to configure and validate S3 bucket lifecycle management in OneFS 9.14 and later releases.

To configure the feature, an S3 command must be sent to the cluster. A simple way to accomplish this without writing code is by using a utility like the ‘s3cmd’ tool. This tool provides a Python script which can be executed directly on a PowerScale cluster.

1. The s3cmd tool’s zip file can be downloaded (or copied) to a directory on the cluster and unpacked with the following CLI command:

# unzip s3cmd-2.4.0.zip

Once unzipped, a new subdirectory named ‘s3cmd-2.4.0’ (with the corresponding version-specific suffix) is created. The working directory should be changed to this new subdirectory so that the ‘s3cmd’ script itself can be executed. The contents of the directory are as follows:

# ls

S3cmd-2.4.0     s3cmd-.2.4.0.zip

#cd s3cmd-2.4.0

# ls

INSTALL.md      NEWS            S3              s3cmd.egg-info

LICENSE         PKG-INFO        s3cmd           setup.cfg

MANIFEST.in     README.md       s3cmd.1         setup.py

2. Next, a test bucket is configured on the cluster, and an access key and secret are generated for that bucket. In this example, the ‘root’ user and the ‘System’ multi-tenant access zone are used for access. This process begins by creating the test directory, verifying that the S3 service is enabled, and disabling HTTPS-only access.

# mkdir -p /ifs/s3lifecycle
# isi s3 settings global modify --service=true --https-only=false

Next, the bucket is created, in conjunction with the access key and secret. For example:

# isi s3 buckets create --name=s3life --path=/ifs/s3lifecycle --owner=root
# isi s3 keys create --user=root --force --show-key > /ifs/root-s3.keys
# isi s3 buckets list
Bucket Name  Path             Owner  Object ACL Policy  Object Lock Enabled  Lock Protection Mode  Description
---------------------------------------------------------------------------------------------------------------
s3life       /ifs/s3lifecycle root   replace            No                   -
---------------------------------------------------------------------------------------------------------------
Total: 1
# cat /ifs/root-s3.keys
       Access ID: 1_root_accid
      Secret Key: 0t1O0URz0H5pef6Wn6P6L9BKc8Ad
       Timestamp: 2026-05-14T14:27:36
  Old Secret Key: ****************************
Old Key Timestamp: 2026-05-12T17:25:02
  Old Key Expiry: 2026-05-14T14:37:36

3. After obtaining the access ID and secret, s3cmd can be configured with these credentials and the appropriate endpoint parameters to simplify command execution. Site-specific configuration parameters that will need to be specified include:

• Access Key
• Secret Key
• S3 Endpoint
• DNS-style bucket+hostname:port

If HTTPS is preferred, the S3 endpoint port should be changed from 9020 to 9021, and Y should be selected when prompted to use the HTTPS protocol. HTTP may be used instead when packet‑level debugging is required, as it allows network sniffing tools such as Wireshark to capture traffic more efficiently and comprehensibly.

# python3 s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 1_root_accid
Secret Key: 0t1O0URz0H5pef6Wn6P6L9BKc8Ad
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 127.0.0.1:9020

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: s3://%(bucket)

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
 Access Key: 1_root_accid
 Secret Key: 0t1O0URz0H5pef6Wn6P6L9BKc8Ad
 Default Region: US
 S3 Endpoint: 127.0.0.1:9020
 DNS-style bucket+hostname:port template for accessing a bucket: s3://%(bucket)
 Encryption password:
 Path to GPG program: None
 Use HTTPS protocol: False
 HTTP Proxy server name:
 HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

4. Utilities such as the ubiquitous ‘dd’ CLI command are useful for easily and rapidly generating some test data files. Similarly, the ‘touch’ CLI command can be used to alter the ‘atime’ (last access) and ‘mtime’ (last modified) timestamps of these files. For example:

# dd if=/dev/zero of=/ifs/s3lifecycle/smlfl1 bs=1k count=1
# dd if=/dev/zero of=/ifs/s3lifecycle/smlfl2 bs=1k count=1
# touch -A -400000 /ifs/s3lifecycle/smlfl2*
# dd if=/dev/zero of=/ifs/s3lifecycle/bigfl1 bs=1M count=1
# dd if=/dev/zero of=/ifs/s3lifecycle/bigfl2 bs=1M count=1
# touch -A -250000 /ifs/s3lifecycle/big*
# dd if=/dev/zero of=/ifs/s3lifecycle/notrmv1 bs=1M count=1
# dd if=/dev/zero of=/ifs/s3lifecycle/notrmv2 bs=1M count=1

# ls -l /ifs/s3lifecycle/

total 4147
-rw-------     1 root  wheel  1048576 May 13 13:48 bigfl1
-rw-------     1 root  wheel  1048576 May 13 13:48 bigfl2
-rw-------     1 root  wheel  1048576 May 14 14:48 notrmv1
-rw-------     1 root  wheel  1048576 May 14 14:48 notrmv2
-rw-------     1 root  wheel     1024 May 12 22:48 smlfl1
-rw-------     1 root  wheel     1024 May 12 22:48 smlfl2
# date
Thu May 14 14:49:26 GMT 2026
 A total of six test files are created, two of which are 1 MB in size and have a modified time more than 1 day prior, two files that are 1 MiB in size but modified within 1 day, and two small 1 KB files with modification timestamps from 2 days prior.

5. Next, add an XML, such as below, into a text file on the cluster. In the following example, the file is named /ifs/lifecycle.xml

<LifecycleConfiguration>
 <Rule>
   <Filter>
      <ObjectSizeGreaterThan>10000</ObjectSizeGreaterThan>
   </Filter>
   <Status>Enabled</Status>
   <Expiration>
     <Days>1</Days>
   </Expiration>
 </Rule>
</LifecycleConfiguration>

Once the XML request body has been added to the lifecycle.xml file, the ‘s3cmd’ can be run to ‘put’ the policy on the bucket. Once done, run a get lifecycle command to confirm that the policy has been applied correctly.

# python s3cmd setlifecycle /ifs/lifecycle.xml s3://s3life
s3://s3life/: Lifecycle Policy updated
onefs914-1# python s3cmd getlifecycle s3://s3life
<?xml version="1.0" ?>
<LifecycleConfiguration>
       <Rule>
               <Status>Enabled</Status>
               <Expiration>
                       <Days>1</Days>
               </Expiration>
               <Filter>
                       <ObjectSizeGreaterThan>10000</ObjectSizeGreaterThan>
               </Filter>
       </Rule>
</LifecycleConfiguration>

6. Next, compare the data from pre and post S3Lifecycle job runs:

# ls /ifs/s3lifecycle
bigfl1        bigfl2        notrmv1       notrmv2       smlfl1       smlfl2
# isi job start S3Lifecycle
Started job [25]
# ls /ifs/s3lifecycle
notrmv 1       notrmv l2       smlfl1       smlfl2

After the S3Lifecycle job is initiated and allowed to run for a short period, the results show that two of the larger files have been deleted while the remaining files remain unchanged. This behavior is expected and reflects the configured expiration and filter criteria defined in the XML policy, under which only objects larger than 10,000 bytes and older than one day qualify for automatic deletion.

If and when it comes to investigating and troubleshooting S3 lifecycle, the following issues and possible resolutions may be useful:

Issue​	Background Potential Resolution​
Unable to create lifecycle policy on bucket​	Bucket owner must be the same as directory owner​
Objects marked for deletion have not been deleted​	Ensure object or bucket does not have lock protection enabled​
​	Job for deletion has not been run​
​	Job for deletion is still in progress​

Beyond this, S3 operations are logged in the S3 log file ‘/var/log/s3.log’.​ Similarly, S3 Job Engine job operations and deleted objects are logged in ‘/var/log/isi_job_d_s3_lifecycle.log’.

Introduced in OneFS 9.14, the PowerScale S3 Lifecycle Management feature enables policy‑driven object management within PowerScale S3 buckets by supporting the automated deletion of objects based on administrator‑defined criteria such as object age, size, or key prefix. These lifecycle policies are applied consistently to both existing and newly created objects within a bucket. Backend processing is managed by the OneFS Job Engine, which performs daily evaluations of configured lifecycle rules and creates per‑bucket tasks to traverse bucket directories and remove objects that meet the specified conditions.

Newly introduced S3 API support in OneFS 9.14 and later includes the following endpoints:

API Endpoint	Description
PutBucketLifecycleConfiguration	Sets the lifecycle configuration for the bucket and replaces any existing one. User must be the bucket owner to create the lifecycle configuration.
GetBucketLifecycleConfiguration	Returns the current lifecycle configuration for the bucket. User must be the bucket owner to get the lifecycle configuration. Will return a ‘NoSuchLifecycleConfiguration’ error if a configuration is not found.​
DeleteBucketLifecycle	Deletes the lifecycle configuration for the bucket. User must be the bucket owner to delete the lifecycle configuration.

Additionally, the following S3 endpoints are also updated in the OneFS 9.14 release and require the following read and write permissions:

S3 Endpoint	Read Permission	Write Permission
CompleteMultiPartUpload	x	x
CopyObject	x	x
GetObject	x
HeadObject	x
PutObject	x	x

Note that all the above operations have an updated ‘x-amz-expiration’ response header if ‘objectexpiration’ has been configured. For example:

HTTP/1.1 200 OK​

…​

x-amz-expiration: expiry-date="Wed, 30 Apr 2027 00:00:00 GMT",rule-id=“3024"​

Content-Length: 434234​

Content-Type: text/plain

S3 bucket lifecycle behavior In OneFS 9.14 enables the bucket owner to define lifecycle management policies that are enforced using root‑level credentials. Lifecycle processing respects object lock protections, ensuring that immutable objects are preserved and not subject to deletion. Lifecycle rules may define expiration based on either a relative time period measured in days since the object’s last modification or an absolute date and timestamp, and can incorporate filtering criteria based on object size and key prefix, including support for backdated expiration rules. Each bucket supports a maximum of 1,000 lifecycle rules.

Note that the OneFS CLI, WebUI, or platform API do not currently support bucket lifecycle configuration, which can only be performed though the S3 API. Additionally, object tag–based filtering is not supported in OneFS 9.14, nor are object transition policies, ‘ExpiredObjectDeleteMarker’ configurations, noncurrent version expiration or transition rules, or versioned objects.

Under the hood, the core S3 bucket lifecycle management architecture is as follows:

At a high level, an S3 client sends a request containing the desired lifecycle rule(s), and the cluster’s S3 protocol head saves that into OneFS’ Tardis configuration database. The OneFS Job Engine retrieves the lifecycle configuration and then proceeds to walk the bucket and file structure, deleting files based upon the expressed rules.

The lifecycle processing is handled by the ‘S3Lifecycle’ job, which, by default, runs daily at 1:00 AM. Scheduling and priority for this job are optionally configurable through the OneFS CLI by a privileged local user:

# isi job types view S3Lifecycle

         ID: S3Lifecycl

Description: Manage S3 object lifecycle per bucket lifecycle policy.

    Enabled: Yes

     Policy: LOW

   Schedule: every day at 1:00am

   Priority: 6

During execution, the job logs all deleted objects and generates an S3Lifecycle job report, which can be viewed with the following OneFS CLI command:

# isi job reports view <job id>

Job report output is along the lines of the following:

S3Lifecycle[22] phase 1 (2026-04-30T11:18:06)​

---------------------------------------------​

Files 3​

Directories 1​

Apparent size 5300​

Physical size 52224​

Objects Deleted 2​

Objects Evaluated 3​

Objects No Action 1​

JE/Error Count 0​

JE/Time elapsed 3 seconds​

JE/Time working 3 seconds​

​

S3Lifecycle[22] Job Summary​

---------------------------​

Final Job State Succeeded​

Phase Executed

As above, this report summarizes the number of objects deleted and skipped, and logs detailed information about all deleted objects to /var/log/isi_job_d_s3_lifecycle.log.

The following table outlines the parameters that are supported for use within the lifecycle XML configuration body.

Configuration​	Description​
ID​	Unique identifier for the rule​.
Status​	If ‘Enabled’, the rule is currently being applied. If ‘Disabled’, the rule is not currently being applied. This is a mandatory field.​
Expiration​	​
Date​	Specifies expiration of the object in the form of date timestamp.​
Days​	Specifies expiration of the object in the form of days​.
Filter​	​
ObjectSizeGreaterThan​	Minimum object size to which the rule applies​.
ObjectSizeLessThan​	Maximum object size to which the rule applies​.
Prefix​	Prefix identifying one or more objects to which the rule applies.​
And​	Apply a logical ‘AND’ to two or more rules inside the operator​.

The ‘PutBucketLifecycleConfiguration’ request takes the following form:

PUT /?lifecycle HTTP/1.1​

Host: Bucket.s3.amazonaws.com​

x-amz-expected-bucket-owner: ExpectedBucketOwner​

<?xml version="1.0" encoding="UTF-8"?>​

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">​

    <Status>Enabled</Status>​

        <Rule>​

            …​

        </Rule>​

        <Rule>​

            …​

        </Rule>​

</LifecycleConfiguration>

As for the rules themselves, these can be temporal in nature. For example, expressing expiration as a date and/or time box:

<Rule>​

    <Expiration>​

    <Date>2029-03-02T12:30:00</Date>​

    <Days>365</Days>​

    </Expiration>​

</Rule>

The ‘date’ tag can be used to specify a time stamp, so objects in the bucket that exist during that date expression (for instance, 12:30 on March 2nd, 2029 in the example above) will be deleted. The ‘date’ tag can include a date in the past. In addition to an explicit date, a rule can also include a ‘days’ tag, which, in the example above, targets objects that haven’t been modified in 365 days, marking them for deletion.

Rules can also specify maximum and/or minimum object size in bytes. For example, greater than 1KB (1024) bytes but less than 50KB (51200 bytes):

<Rule>​

    <Filter>​

    <ObjectSizeGreaterThan>1024</ObjectSizeGreaterThan>​

    </Filter>​

<Rule>​

</Rule>​

    <Filter>​

    <ObjectSizeLessThan>51200</ObjectSizeLessThan>​

    </Filter>​

</Rule>​

Rules filters can also specify data locality. For example:

<Rule>​

<Filter>​

    <And>​

        <ObjectSizeGreaterThan>18124</ObjectSizeGreaterThan>​

        <ObjectSizeLessThan>92686</ObjectSizeLessThan>​

<Prefix>/data/path/</Prefix>​

    </And>​

</Filter>​

</Rule>

Note that, when specifying more than one filter, they must be wrapped in the <And> element, as above.

A bucket’s lifecycle configuration can be queried with the ‘GetBucketLifecycleConfiguration’ request. For example:

GET /?lifecycle HTTP/1.1​

Host: Bucket.s3.amazonaws.com​

x-amz-expected-bucket-owner: ExpectedBucketOwner

In contrast, a lifecycle can also be removed with the ‘DeleteBucketLifecycle’ request. For example:

DELETE /?lifecycle HTTP/1.1​

Host: Bucket.s3.amazonaws.com​

x-amz-expected-bucket-owner: ExpectedBucketOwner

In addition to bucket lifecycles, OneFS 9.14 also provides lifecycle management for incomplete S3 MPU operations. Specifically, this is the ability to craft lifecycle rules which clean up incomplete multipart uploads, helping reclaim space from abandoned large object transfers.

Configuration​	Description​
AbortIncompleteMultipartUpload​	​
DaysAfterInitiation​	Number of days after the system aborts an incomplete MPU​.

Rule parameters include the number of days after MPU initiation and the data path prefix. For example:

<Rule>​

    <AbortIncompleteMultipartUpload>​

        <DaysAfterInitiation>7</DaysAfterInitiation>​

    </AbortIncompleteMultipartUpload>​

<Filter>​

        <Prefix>/path/to/data/</Prefix>​

    </Filter>​

</Rule>

Note that the ‘DaysAfterInitiation’ parameter is limited in scope to incomplete multipart uploads (MPUs) and does not apply to standard objects or completed MPU uploads, while the lifecycle expiration rules similarly exclude incomplete MPUs.

In the next article in this series, we’ll look at some practical examples of how to configure, use, and validate S3 bucket lifecycle management in OneFS 9.14 and later releases.

PowerScale OneFS inline compression is a native data reduction feature that operates in the write path to improve storage efficiency by compressing data before it is written to disk. This helps OneFS drive data reduction efficiencies to support PowerScale’s contract-backed 2:1 data reduction ratio (DRR) guarantee. Plus, many workloads and deployments achieving substantially higher ratios in practice, including EDA up to 6.5:1 compression and life sciences up to 4:1 compression.

As part of a multi-stage pipeline that includes zero-block elimination and inline deduplication, OneFS evaluates data in 128 KB regions and selectively compresses those that achieve useful space savings, storing them as compact encoded containers while leaving incompressible data unmodified.

Because this process occurs inline, it not only influences capacity utilization, but also CPU consumption, write latency, and on-disk data layout. When used with suitably compressible workloads, inline compression can significantly reduce physical I/O and increase effective cluster capacity. However, its impact on performance and data placement calls for understanding and careful consideration, particularly for workloads with low compressibility or frequent small updates.

Inline compression is often a beneficial optimization, with modest additional CPU overhead reducing physical I/O demands while increasing effective storage capacity. For many workloads, this expectation holds true. However, for incompressible data sets, enabling OneFS inline compression can negatively impact performance as compared to running with no compression at all, and the penalty doesn’t abate instantly upon disabling the feature.

So for data such as encrypted backups, already-compressed media, highly random content, or when randomly overwriting data that was previously written in a compressed layout, performance can degrade in ways that are predictable once the underlying on-disk activities are understood. The crux is that inline compression is not just a logical efficiency feature. It changes how data is laid out and how writes are handled. That means the impact is not limited to capacity savings, since it can also influence write throughput, overwrite behavior, and the cost of small updates into existing files.

Without inline compression, OneFS stores data in 8 KB blocks, grouped into protection groups (e.g. 4+2 on a six node cluster/pool with a +2n FEC protection policy).

A 128 KB logical range (stripe unit) comprises 16 × 8 KB blocks laid out as plain data or parity blocks. Conversely, with inline data reduction enabled, every incoming write flows through a multi-stage pipeline before it ever touches disk:

Order	Stage	Description
1	Zero block removal	Fully-zero 8 KB blocks are detected and stored as sparse references rather than real blocks.
2	Inline dedupe	8 KB blocks are fingerprinted and matched against an in-memory hash table; duplicates are collapsed to shared references.
3	Inline compression	Surviving 128 KB regions are handed to the compression engine (zlib or lz4 depending on platform), which decides per-region whether the result is small enough to keep.
4	Protection and layout	Whatever comes out the other side (compressed container, deduped reference, or plain blocks) is then protected with FEC and laid out across the cluster.

Note that every stage in this pipeline consumes cluster resources (CPU, etc) and adds write-path latency on every write – before OneFS knows whether it will deliver a benefit. Two things change on disk as a result:

1. Data is chunked into 128 KB compression regions. Each region maps to 16 logical 8 KB blocks, and this is what the compression engine actually works on.
2. Compressible regions are stored as compressed containers. When a 128 KB region compresses well enough, OneFS encodes it into a smaller bytestream using a node-specific algorithm (zlib, lz4, etc., fixed per platform and OneFS release). That region is no longer stored as 16 individual blocks, instead becoming a single compressed container.

Compressed regions are reported via the ‘isi get -DDO’ CLI command, as follows:

lbn 160: 10+2/2

...

5,5,1403920384:8192[DIRTY,COMPRESSED]#2

(sparse)[DIRTY,COMPRESSED]#3

...

{ebns={160,161},lbns={160,161,162,163,164},

encoded_size=10762,encoded_offset=0,

decoded_blocks=5,format=zlib}

The ‘encoded_size’ parameter returns the compressed byte count while ‘decoded_blocks’ reports how many logical 8 KB blocks live inside. The file looks identical from the outside, but, physically, the layout is where things get interesting since OneFS doesn’t know data is incompressible until it attempts to compress it. So even for data that will never compress, the engine still:

1. Takes the 128 KB chunk.
2. Runs it through the compression algorithm.
3. Checks if compressing the chunk does save at least one full 8 KB block.
4. Falls back to writing it as a chunk of normal, uncompressed 8 KB blocks.

On disk, those blocks show up tagged as ‘INCOMPRESSIBLE’. For example:

lbn 0: 10+2/2

...

6,2,1176543232:8192[DIRTY,INCOMPRESSIBLE]#16

...

1,5,1075085312:8192[DIRTY,INCOMPRESSIBLE]#12

No {encoded_size=…, decoded_blocks=…, format=…} stanza — just plain blocks. But the cluster still expended CPU and latency as a cost of trying. That overhead is real, and on CPU-constrained A-series archive nodes, it can show up directly in throughput.

That said, OneFS inline compression can be easily disabled entirely via the following CLI syntax:

# isi compression settings modify --enabled=False

With the ‘–enabled’ parameter set to ‘False’, new writes automatically skip the compression stage completely. Each 128 KB region goes straight to disk as plain 8 KB blocks — no attempt, no fallback, no overhead. The on-disk layout looks identical to the incompressible case above (without the INCOMPRESSIBLE tag), but the ‘attempted compression’ overhead never affected the write path.

However, note that disabling compression is not retroactive. Disabling compression afterwards doesn’t decompress or rewrite (re-lay-out) those regions in the background. That’s why you can disable compression cluster-wide and still see what looks like ‘compressed behavior’ on overwrites to older data. These regions can be identified by their ‘[COMPRESSED]’ tags and the compression metadata block in output from the ‘isi get -DDO’ CLI command.

With compression enabled, every write still flows through the full pipeline (zero-block removal > dedupe > compression) regardless of whether the data ends up compressed or uncompressed on disk. Hence, the latter case incurs  CPU overhead and write-path latency for no additional space savings.

On platforms like the PowerScale A310, this translates to a measurable throughput drop for workloads dominated by encrypted backups, already-compressed files, or just very random data. Disabling compression and re-running against fresh data will often result in write throughput increasing noticeably.

Note that the penalty for incompressible data is often more significant than for genuinely compressible or dedupe-able data. When data compresses or deduplicates well, inline data reduction actually reduces the amount of physical I/O hitting the drives, due to fewer blocks to write, less parity to compute, less backend traffic. The CPU overhead is more than offset by the I/O savings, resulting in better throughput and lower latency than without data reduction at all. In contrast, uncompressible data yields none of that I/O relief, but only the overhead.

When over-writing existing compressed data, a compressed region is a 128 KB atomic unit. This means that even a 4 KB update in the middle of a region will force OneFS to:

1. Read the full 128 KB compressed container.
2. Decompress it in memory.
3. Merge the changed bytes.
4. Re-encode and rewrite the full 128 KB region.

This read–decompress–modify–rewrite (R-DM-W) penalty applies even with inline compression disabled compression. If the region on disk was written compressed, the overwrite path still must treat it that way until the full file is physically rewritten with an uncompressed layout.

Even over-writing regions that are ‘incompressible’ incur an overhead. Because a file written with compression enabled may contain a mix of compressed and uncompressed (incompressible) regions, OneFS has to route partial-chunk overwrites through the compression stage to stay consistent across the 128 KB boundary. It can’t know ahead of time whether a given 128 KB range is fully uncompressed, so it has to treat the whole chunk conservatively.

This means that even ‘incompressible regions (i.e. data that never compressed in the first place) still carry a partial-overwrite penalty if compression was enabled when the file was initially written. Only a full 128 KB chunk replacement can safely bypass the compression engine entirely for that region, which is usually achieved by overwriting the file completely. This often manifests itself as:

• New writes returning to full speed immediately upon disabling compression.
• Overwrites remaining sluggish until the underlying data is physically rewritten as uncompressed — regardless of whether the on-disk blocks are tagged ‘[COMPRESSED]’ or ‘[INCOMPRESSIBLE]’.

Investigating and troubleshooting unexpected performance with inline compression typically involves:

1. First, confirming whether the data is actually compressible by verifying data reduction statistics at the cluster or path level:

# isi statistics data-reduction view

# isi compression stats view

# isi_storage_efficiency /ifs/path/to/file

If compression is ‘enabled’ but little to no space savings are evident, the data almost certainly isn’t compressible.

2. Next, check the actual on-disk layout:

# isi get -DDO /ifs/path/to/file

Specifically, the ratio of ‘[COMPRESSED]’ to ‘[INCOMPRESSIBLE]’ tags in the ‘PROTECTION GROUPS’ section, and the ‘Metatree logical blocks:’ summary at the end.

3. Separate new writes from overwrites when testing:

This is the most important thing to get right when benchmarking. Use fresh test files written after any compression settings reconfiguration, and run tests in two distinct phases:

• Phase 1 — Initial write (e.g. –overwrite=0 in fio)
• Phase 2 — Overwrite / update (e.g. –overwrite=1 with random updates into the same files)

The following is a simple example ‘fio’ test overwriting an existing file at random offsets:

fio --name=random-update \

--filename=data.bin \

--rw=randwrite \

--bs=1M \

--size=10G \

--direct=1 \

--overwrite=1 \

--ioengine=libaio \

--iodepth=64

Then compare:

Test	What it reveals
Initial writes: compression on vs. off	The cost of the “try to compress” overhead
Overwrites into files written with compression on	The R-DM-W penalty on legacy compressed regions
Overwrites into files written with compression off	Your true baseline uncompressed overwrite performance

4. Rule out unrelated bottlenecks.

Before assuming a compression issue, validate that the cluster is not hitting CPU, disk, or network limits for other reasons:

# isi statistics system list --nodes=all --format=top

# isi statistics drive list --format=top --sort=Queued

# isi statistics client list --format=top

# isi statistics protocol list --format=top

The goal is to distinguish between hitting the limits of the compression design from saturating the hardware. Having confirmed that legacy compressed layout is hurting performance, disabling compression, then physically rewriting the data is generally the cleanest approach. The following options can be used to ‘rehydrate’ the data back to a regular uncompressed layout.

1. Disable compression globally (or via file pool policy)

# isi compression settings modify --enabled=False

2. Or rewrite via intra-cluster copy using the ‘cp’ CLI command:

# cp -a /ifs/source/path /ifs/target/path

3. Use SyncIQ or SmartSync to replicate/copy to another /ifs location within the cluster.

Once rewritten, the ‘isi get -DDO’ CLI will show only uncompressed blocks, and overwrites follow the efficient block-level path rather than a read-decompress-modify-write (R-DM-W) operation.

4. If rewriting from the application is impractical, the SmartPoolsTree job can be used instead. Note that this method requires SmartPools to be licensed across the cluster, and the job settings configured as follows:

a) Set the SmartPoolsTree job to a ‘retune’ restriping strategy via gconfig:

# isi_gconfig -t job-config jobs.types.smartpoolstree.find_goal=retune

# isi_gconfig -t job-config jobs.types.smartpoolstree.restripe_goal=retune

b) Then run against the target path

# isi job start SmartPoolsTree --paths=/ifs/path/to/data

Or, for a more direct, non-job-engine approach, use isi set (no SmartPools license required):

# isi set -r -g retune /ifs/path/to/data

5. Alternatively, if SmartPools is unlicensed or using the job engine is undesirable, for a more direct, the ‘isi set’ CLI command can be used as follows:

# isi set -r -g retune /ifs/path/to/data

Note the case sensitivity of the ‘isi set’ ‘-r’ and ‘-R’ flags: The lowercase ‘-r’ flag forces an immediate restripe of the file using the specified goal (-g retune), effectively decompressing and re-laying-out the data without going through the job engine. The uppercase ‘-R’ flag applies the goal recursively to an entire directory tree, which can be an extremely time-consuming operation with little visibility or control over speed and duration. Scripting ‘isi set -r -g retune’ to iterate over a list of hot files or a specific subdirectory is typically preferred, providing improved control. Because this operation trades background I/O and capacity (by giving back compression savings), it’s best targeted at specific uncompressible or otherwise hot datasets rather than an entire archive tier.

Inline data reduction in OneFS can be highly beneficial for the appropriate workloads, but its underlying on-disk behavior can lead to unexpected outcomes if unfamiliar with its operation. Key takeaways include:

• Incompressible data still incurs a compression cost when compression is enabled: CPU resources are consumed and write-path latency increases even when no actual space savings occur. In some cases, this overhead can be more impactful than with compressible data. When data compresses or deduplicates effectively, reduced I/O often offsets CPU usage and can even improve performance. With incompressible data, however, there are no such benefits—only added overhead.
• Both compressed and incompressible regions are subject to read–decompress–modify–write (R-DM-W) overhead on every overwrite, regardless of the current compression setting.
• Turning off compression benefits new writes right away, but it does not change the layout of data that was already written in a compressed format.
• Data originally written as incompressible continues to experience 128 KB partial-chunk overhead during partial overwrites until those areas are either fully overwritten or the entire file is replaced.
• A solid understanding of how OneFS implements data reduction—and a clear view of your workload characteristics before enabling it—helps avoid these pitfalls entirely.

As with many things in life, compression usage is a cost-benefit conundrum. Once the underlying mechanics are understood, their behavior is consistent and predictable. The key is to evaluate, measure, and quantify, and apply the knowledge and findings proactively, rather than reacting after performance issues arise in production.

The previous article in this series provided an overview of the S3 multipart upload (MPU) functionality in PowerScale OneFS. Now, we’ll turn our attention to extended MPU functionality introduced in OneFS 9.13, which allows PowerScale S3 users to track the status and completion progress of their MPU completion in real time.

MPU Completion is an existing standard S3 API that has been extended to return three additional response headers, Completion‑Id, Fast‑Path, and Completion‑State‑Info, when MPU completion progress tracking is enabled. MPU Completion Progress is a newly added extended S3 API that reports real‑time progress information in the response, including whether the fast path is used, the total number of parts and recovery parts, and the counts of completed and recovered parts.

Disabled by default, MPU completion progress activation on a PowerScale cluster is controlled by the boolean ‘MPUCompletionProgressEnable’ gconfig parameter, which must be set to value ‘1’, followed by a restart of the S3 service, for it to take effect. The CLI syntax for this procedure is as follows:

# isi_gconfig registry.Services.lwio.Parameters.Drivers.s3.MPUCompletionProgressEnable=1

# isi services s3 disable

# isi services s3 enable

Starting with OneFS 9.13, multipart upload (MPU) completion can follow either a fast or slow path based on part integrity and ordering, with fast‑path completion occurring when all parts are the same size except possibly the last, parts are retransmitted only in the event of failure while preserving their original sizes, every uploaded part is included in the final object, and part numbers are contiguous beginning at 1 with no gaps. When these conditions are satisfied, OneFS performs a highly optimized assembly process. If any condition is violated, OneFS falls back to the slow path, which requires more processing time.

OneFS 9.13 and later also include the following completion progress API endpoint:

GET /{bucket}/{object-key}?completionId=<completionId>&uploadId=<uploadId>

When progress tracking is enabled, the ‘CompleteMultipartUpload’ response includes a ‘completionID’, which is especially valuable for very large uploads where completion can take a noticeable amount of time, as it allows clients to determine whether the operation is using the fast or slow path, view the total number of parts involved, and retrieve real‑time completion metrics through the ‘Completion‑State‑Info’ data.

The components marked in green in the following diagram are introduced in OneFS 9.13, with ‘CompleteMultipartUpload’ used to assemble all the uploaded parts, and ‘GetMPUCompletionProgress’ to check the progress.

Component	Action
S3 Protocol Head	S3 Protocol Head processes requests from S3 clients.
Likewise Iomgr	Likewise Iomgr provides IO APIs. Both MPU requests need to call the Iomgr in order to read and write progress status file(s).
FS Layer	File system layer performs IO requests from upper layers.
Gconfig	Gconfig is in charge of storing user configuration to enable or disable this feature.
Other	The other components are mainly responsible for processing the two requests from S3 clients.

The following table details the various completion states:

Scenario	Message	Progress
Super fast path	Super fast path with parts 1..N sequential, will finish quickly and completionID will be useless.	No need to check
Fast path	Fast path with parts 1..N sequential, but x parts need to be recovered. May fallback to slow path in case of errors.	To check recovery progress
Slow path: only one part	Not a fast path due to only one part in total and this part needs to be recovered.	To check slow path progress
Slow path: part number not contiguous	Not a fast path due to non-contiguous part numbers. Please verify intent, or abort and restart the session.	To check slow path progress
Slow path: part number not from 1	Not a fast path due to part numbers not starting from 1. Please verify intent, or abort and restart the session.	To check slow path progress
Slow path: different part size	Not a fast path due to the size of non-last parts differs from the 1st part size. Please verify intent, or abort and restart the session.	To check slow path progress

For example, the following shows the HTTP 200 response headers from a fast path MPU operation on a large file:

Pertinent information returned includes:

Attribute	Example Output
completion-state-info:	Fast path with parts 1..N sequential, but 10 parts need to be recovered. May fallback to slow path in case of errors.
completion-id:	82d4597e-cb87-8a21-2087-a8c6163632b1
fast-path:	true

For non‑MPU objects, OneFS can compute MD5-based ETags when configured (use Md5 for Etag and Validate Content Md5).
Note that objects created via multipart upload do not use a simple MD5 hash as the ETag.
As with AWS S3, MPU ETags may be composite values, and applications requiring MD5 semantics must account for this.

When it comes to investigating and troubleshooting S3 MPU issues, the following error response codes can be helpful:

Status	Description	Troubleshooting
400 Bad Request	Unexpected upload Id Unexpected completion Id format	Check if  upload ID is correct. Check if completion id is correct (it can be got from CompleteMPU response header), cannot be blank.
403 Forbidden	No correct permission.	Only bucket owner, MPU completion initiator, or users who have been granted READ/FULL_Control permissions on the bucket can do this operation.
404 NoSuchCompletionId	Failed to find MPU complete progress.	Check if the completion has already done. Check if completion ID is correct (ID can be found in the CompleteMPU response header)

 

It’s also worth nothing that Multipart upload is distinct from SigV4 streaming (chunked upload, available in OneFS 9.3 and later), which controls how payloads are signed and transmitted rather than how objects are partitioned. SigV4 chunked transfer encoding can be used with or without MPU.

Comparison:

Characteristic	Multipart Upload	SigV4 Chunked Upload
Purpose	Breaks object into logical parts	Breaks HTTP payload into signed chunks
Identifiers	uploadId, part numbers	SigV4 per‑chunk signatures
Relationship	Independent mechanisms	Can be used together

 

So, in summary, OneFS S3 MPU follows the same client-facing operation and semantics as AWS S3:

CreateMultipartUpload → UploadPart / UploadPartCopy → CompleteMultipartUpload / AbortMultipartUpload, plus listing operations.

Internally, OneFS stores multipart upload segments in ‘.s3_parts_<uploadId>’, assembles those parts into the final object upon completion, and removes the part files after either completion or abort, while OneFS 9.13 and later add MPU enhancements such as fast‑ or slow‑path determination for completion and real‑time progress monitoring via ‘GetMPUCompletionProgress’, including response headers like ‘Completion-Id’ and ‘Fast-Path’.