PowerScale OneFS is a distributed operating system, and keeping a cluster of up to 252 nodes in a consistent, well-managed state requires a clear division of responsibility across a set of foundational infrastructure components. Six of these underpin almost everything that happens in a cluster: gconfig, Tardis, MCP, the Job Engine, Persistent Queue (PQ), and sysctls.
Most of these components have been present across many OneFS generations, but they are frequently referenced in documentation without much explanation of exactly how they relate to each other.
- At the bottom of the stack sits gconfig, OneFS’ distributed, replicated key-value configuration store. Every node in the cluster maintains a replica, and changes are propagated and committed cluster-wide before they take effect. gconfig holds raw cluster state: node membership, quorum data, join queues, distributed locks, and low-level service flags. It is the substrate everything else builds on — not because it is the most user-facing component, but because it is the one that has to survive partial cluster availability, node failures, and split-brain scenarios.
A good example of gconfig in direct use is the Automated Cluster Setup (ACS) system, which queues pending node serial numbers into gconfig under the gcfg_lazy_join.queued_nodes key. The Isi Lazy Join (ILJ) coordinator thread reads these entries directly from gconfig, attempts to join each node via the IPT discovery protocol, and then removes successfully joined entries from the queue. That entire flow (queue, match, join, dequeue) runs through gconfig without any higher-level abstraction layer in between. The same pattern applies to distributed lock coordination and quorum state management.
- Many newer OneFS features do not write to gconfig directly. Instead, they go through Tardis. But gconfig remains the authority for the lowest-level cluster mechanics (i.e. the things that have to work even when higher-level services have not yet started).
Tardis is the configuration service and database that sits above gconfig. Where gconfig is a raw key-value store accessed with low-level calls, Tardis presents a structured, service-oriented interface that higher-level components use to read and write their configuration. When a user submits a configuration change through the OneFS CLI, WebUI, or the platform API (pAPI), the pAPI handler writes the new configuration into Tardis, and Tardis notifies the relevant subscriber services.
The range of features that use Tardis is broad. CELOG uses it for event, alert, and channel configuration. S3 Lifecycle Management stores its bucket rules there. The Job Engine reads job type schedules, priorities, and impact policies from Tardis. NFS, SMB, and other protocol services store their configuration through pAPI into Tardis. In each case, the flow is the same: The pAPI handler receives the request, validates it, and writes the authoritative configuration into Tardis. The relevant daemon then picks up the change via its subscription to the Tardis notification stream.
- MCP, the Master Control Program, is OneFS’s service lifecycle manager — its equivalent of a system init daemon. Every userspace daemon that runs on a PowerScale node is started, monitored, and restarted by MCP. This includes isi_job_d, isi_papi_d, isi_stats_d, isi_nfs_d, and dozens of others. MCP watches both gconfig and Tardis for configuration changes that affect service state, and responds accordingly. If a feature is enabled that requires a new daemon, MCP starts it. If a daemon crashes, MCP restarts it, respecting any configured restart delay or retry limit.
MCP also enforces service dependencies. isi_job_d cannot start until gconfig and Tardis are both available. The platform API cannot start until lower-level services are ready. This sequencing is critical during node boot: the order in which services come online determines what is and is not available during each phase of startup. Getting that order wrong results in services that start without their dependencies satisfied, which leads to failures that are often confusing to diagnose.
From a troubleshooting perspective, MCP is often the first place to look when a daemon is unexpectedly absent or restarting. The ‘isi services’ CLI command displays MCP-managed service state across the cluster:
# isi services -a | grep -i job isi_job_d OneFS Job Engine Enabled
- Sysctls are FreeBSD kernel tunables that control low-level OS and OneFS kernel module behavior. They sit at the boundary between userspace and the kernel — the place where configuration crosses from daemon-managed state into the kernel subsystems that handle networking, memory, I/O scheduling, and protocol processing. In OneFS, sysctls fall into two categories: those that are set per-node at boot and remain static until a reboot or explicit change, and those that are set dynamically by MCP-managed daemons in response to gconfig or Tardis changes.
A concrete example of the dynamic path: the NFS performance diagnostic statistics feature introduced in OneFS 9.15 exposes a slow operation logging threshold via a registry key (NfsSlowOpThresholdMs). That configuration is written through pAPI into Tardis, picked up by the NFS service, and applied as a kernel-level sysctl that the kernel NFS stack then enforces on every request. The user sees a single CLI command; behind it, the change traverses PAPI → Tardis → daemon → sysctl before it has any effect.
Cluster-wide sysctl propagation is coordinated through gconfig, ensuring that every node uses the same value. If needed, per-node overrides can be applied locally but not propagated. This makes sysctls the most ‘node-local’ of the six components since their state lives in the kernel of each individual node, rather than in a shared store.
- The OneFS Job Engine is the distributed background task execution framework responsible for dividing large cluster-wide operations (eg. FlexProtect, SmartPools, MultiScan, MediaScan, S3Lifecycle,etc) into work items and executing them in parallel across all nodes. It runs as a daemon (isi_job_d) on every node, is started and monitored by MCP, and operates as a four-level delegation hierarchy.
At the top of the hierarchy is the coordinator, which is a single process that can run on any node in the cluster. The coordinator is responsible for launching jobs, dividing work, controlling worker thread counts, and processing results as they come in. If the coordinator process dies, coordinator responsibility automatically moves to another node, with the election being handled via a gconfig distributed lock. The node currently acting as coordinator can be identified at any time:
# isi job status --verbose | grep Coordinator Coordinator: node3 (lnn 3)
Below the coordinator, each node runs a director process that acts as a relay between the coordinator and the manager processes on that node. Each running job spawns its own manager on every participating node, and each manager controls a pool of worker threads that process the actual work items. This hierarchy means the coordinator never talks directly to individual workers, so instructions flow down through directors and managers, keeping the coordinator’s communication load manageable even on very large clusters.
The Job Engine reads its configuration (i.e. job type definitions, schedules, priorities, and impact policies) from Tardis. Job state and phase checkpoints are written to gconfig after each phase completes, which is what allows a paused or interrupted job to resume from where it left off even after a coordinator failover or node loss.
Scheduling is managed by the Job Engine’s own internal priority queue. Every job type is assigned a priority from 1 (highest) to 10 (lowest). Up to three jobs can run simultaneously, and the priority queue governs what happens when that limit is reached or when jobs share an exclusion set (a group of job types that cannot run concurrently because they contend for the same resources). When a higher-priority job needs to run and no slot is free, the lowest-priority running job is preempted: it is paused and returned to the back of the Job Engine priority queue, where it waits until it reaches the front again and a slot is available. For jobs of the same type and same priority, the one that was queued first runs first. Scheduling is also governed by cluster health: certain jobs are restricted based on device availability or degraded states. The FlexProtect and FlexProtectLIN jobs carry the highest default priority of 1, reflecting their role in reprotecting data:
# isi job types view FlexProtect ID: FlexProtect Description: Reprotects user data and data on unavailable devices. Enabled: Yes Policy: LOW Schedule: when_source_modified Priority: 1
- Persistent Queue (PQ) is a OneFS infrastructure service that provides ordered, durable queues for both user-mode and kernel-mode consumers, with each queue backed by an SBT (System B-Tree). This is the same B-tree structure that underpins other core OneFS metadata stores, allowing queue entries to survive process crashes, node reboots, and partial cluster failures. Entries are indexed by 128-bit B-tree keys and can carry an opaque data payload of up to approximately 8 KB each. Enqueuing an entry is transactional: either the entry is durably committed to the SBT or the operation fails cleanly. Contention avoidance between concurrent readers and writers is intentionally left to the caller. As such, PQ provides the durable storage primitive, not the concurrency policy.
PQ’s dual user-mode/kernel-mode accessibility is significant. Kernel-side components (eg. protocol handlers, the NFS layer, low-level filesystem operations) can enqueue work items directly without round-tripping through a userspace daemon. Userspace services read from the same queues. This makes PQ a natural integration point between kernel events and userspace processing pipelines, used wherever a kernel subsystem needs to hand off durable work to a userspace consumer without coupling them tightly in time.
A concrete example of PQ in practice is the trash directory queue, where each entry carries a root LIN (Logical Inode Number) and associated metadata. Over time, entries in this queue can become stale when the underlying LIN no longer exists in the filesystem, typically because the directory was already deleted or the data was cleaned up through another path. These orphaned entries illustrate a key property of the PQ design: entries are durable and persist until explicitly removed, and since contention avoidance is the caller’s responsibility, no background process automatically purges them.
PQ addresses this with two operational modes. In dump mode, it enumerates all entries in the trash_directory PQ, either printing all entries or only the stale ones. In cleanup mode, it removes every stale entry from the queue. All operations are logged to /var/log/isi_trash_pq_clean.log and via syslog.
A useful way to witness all six of these OneFS components interacting is to trace two common scenarios from start to finish.
When a cluster admin makes a configuration change (eg. modifying the S3 lifecycle rules for a bucket) the platform API (pAPI) handler receives the request, validates it, and writes the new configuration into Tardis. Tardis notifies the relevant subscribers. If the change affects a service’s lifecycle (eg. enabling/disabling a feature), MCP responds by starting or stopping the appropriate daemon. If the change has a kernel-level enforcement requirement, such as an NFS tunable or a network stack parameter, the daemon applies it via a sysctl. Any derived cluster state (distributed flags, join queue entries) goes directly to gconfig.
On the job execution side, when a job such as S3Lifecycle’s scheduled timeslot arrives, Tardis surfaces that trigger to the Job Engine coordinator. The coordinator checks its internal job scheduling logic, and if fewer than three jobs are running and no exclusion set conflicts exist, the job starts immediately. If not, the coordinator holds the job until a slot opens. Once running, the coordinator divides the bucket tree into work items using the Job Engine’s shared work distribution model, distributing them through directors and managers to worker threads across all nodes. Phase completion is checkpointed to gconfig. If the coordinator node fails mid-job, a new coordinator is elected and reads the last checkpoint from gconfig to resume from the correct phase.
During a node boot, the kernel starts and loads its sysctls from ‘isi_sysctl.cfg’. MCP starts early and reads gconfig to determine service topology. MCP starts Tardis (the configuration service), then pAPI, then isi_job_d and all other feature daemons in dependency order. By the time the node is fully online, all six components are operational, each in its correct layer of the stack.
The division of labor between these components is deliberate. Gconfig handles the things that must work even in degraded cluster states. Tardis provides the structured configuration interface for features that assume the cluster is healthy. MCP ensures daemons are running and in the right state. The Job Engine distributes and executes background tasks across the cluster using its own internal shared work model, with checkpoints persisted to gconfig. PQ provides durable SBT-backed queue storage for other subsystems that require transactional, crash-safe queuing from user or kernel mode. Sysctls enforce configuration at the kernel level, where userspace cannot reach.
In summary, each layer does precisely what it needs to do, and, in concert, they cover the full spectrum from kernel tunables through distributed service lifecycle to long-running background task orchestration.