As we saw in the previous article in this series, at a high level, there are three main elements to a OneFS quota:
| Element | Description |
| Domain | Define which files and directories belong to a quota |
| Resource | The quantity being limited |
| Enforcement | Specify the limits and what actions are taken when those thresholds are exceeded |
We’ll look at each of these elements over the course of this series of articles. But first, let’s delve into the architecture.
Under the hood, SmartQuotas hinges on the quota domain and quota database, and the general operational flow is as follows:
Each quota is governed by a OneFS domain, which defines the quota’s scope and includes a set of usage levels, limits, and configuration options. Most of this information is organized and managed by the file system and stored in the quota database (QDB). This database is represented in a B-tree structure, known as the quota tree, and allows both scalability and fast random access. Because of its importance, the quota database is protected at OneFS’ highest metadata level. The quota accounting blocks (QABs) within individual records are protected at the same level as the associated directory.
A quota domain is made up of the following principal parts:
| Component | Description |
| Quota domain key | Where the unique identifier for the domain is stored. |
| Quota domain header (QDH) | Contains various state and configuration information that affects the domain as a whole. |
| Quota domain enforcements | Manages quota limits, including whether they have been reached or exceeded, notification information, and the quota grace period. |
| Quota domain account (QDA) | Handles tracking of usage levels for the domain. The QDA tracks physical, logical, and file resource types for each domain. |
The QDB is a data structure that stores quota domain record (QDR). Resource allocation and governance changes are recorded in the quota operation associated with a transaction, totaled and applied persistently to the QDRs.
Within QDB, a quota domain record stores all configuration and state associated with a domain. The record can be broken down into three components:
| Component | Description |
| Configuration | Fields within quota config, such as whether the domain is a container. Despite the name, this includes some state fields like the Ready flag. |
| Enforcements | A list of quota enforcements, which include the limit, grace period, and notification state. Although the structure is flexible, only three enforcements are allowed and only for a single resource. |
| Account | The quota account for the domain. |
The on-disk format of the QDR is as follows The structure is dynamic, based on the configured enforcements and state of the account, so the on-disk structures look different than the in-memory structures.
Quota domain locks synchronize access to quota domain records in the QDB.
The main challenge for quota domain locks is that the need to exclusively lock a quota domain is not known until the accounting is fully determined. In fact, it may not be until responses from transaction deltas are received before this is reported to the initiator. To address this, Quota Domain Locks use optimistic restarts.
Quota Account Blocks (QABs) enable high-performance accounting using transaction deltas. Since when the quota usage info if viewed it is stale anyway, locking is simplified by using an exclusive domain lock for coherent reads of usage.
Each QAB contains a large number of Quota Accounting records, which need to be updated whenever a particular user adds or removes data from an area of the file system on which quotas are enabled (quota domain). If a large quantity of clients are simultaneously accessing the quota domain, these blocks can become highly contended and a potential bottleneck. Similarly, if a single client (or small number of clients) consistently makes a large number of small writes to files within a single quota, write performance could again be impacted.
To address this, quota accounts have a mechanism to help avoid hot spots on the nodes storing QABs. Quota Account Constituents (QACs) help parallelize the quota accounting by including additional QAB mirrors distributed across other nodes in the cluster.
Configuration is managed through a sysctl, efs.quota.reorganize.qac_ratio , which increases the number of quota accounting constituents. This provides better scalability and reduces latencies on heavy create/delete activities when quotas are used.
Using this parameter, the internally calculated QAC count for each quota is multiplied by the specified value. If a workflow experiences write performance issues, and it has many writes to files or directories governed by a single quota, then increasing the QAC ratio may significantly improve write performance.
The sysctl efs.quota.reorganize.qac_ratio can be reconfigured to its maximum value of 8 from its default value of 1 using the following CLI command:
# isi_sysctl_cluster efs.quota.reorganize.qac_ratio=8
To verify the persistent change, run:
# cat /etc/mcp/override/sysctl.conf | grep qac_ratio efs.quota.reorganize.qac_ratio=8 #added by script
Although increasing the QAC count through this sysctl can improve performance on write heavy quota domains, some amount of experimentation may be required until the ideal QAC ratio value is found. Adjusting the parameter can adversely affect write performance if you apply a value that is too high, or if you apply the parameter in an environment that does not have diminished write performance due to quota contention.
Additionally, OneFS provides a CLI command, which can restripe the QABs to improve their performance.
# isi_restripe_qabs retune
This utility can be run either on demand or periodically to randomly redistribute QABs for all existing quotas. It does this by ignoring the default ‘rebalance’ layout and running a ‘retune’ layout strategy instead, thereby alleviating the performance impact from an imbalanced QAB layout.