Had a couple of recent enquiries from the field regarding SmartQuotas performance. So in this article we’ll explore one of the more obscure tuning parameters of OneFS SmartQuotas.
Under the hood, SmartQuotas quota data is maintained in Quota Accounting Blocks (QABs). Each QAB contains a large number of accounting records, which need to be updated whenever a particular user adds or removes data from the quota domain, the area of the filesystem on which quotas are enabled. If a large quantity of clients are simultaneously accessing the quota domain, these blocks can become highly contended and a potential bottleneck. Similarly, if a single client (or small number of clients) consistently makes a large number of small writes to files within a single quota, write performance could again be impacted
To address this, quota accounts have a mechanism to help avoid hot spots on those nodes which are storing QABs. This can be addressed using Quota Account Constituents, or QACs, which help parallelize the accounting. QACs can boost the performance of quota accounting by creating additional QAB mirrors, which are distributed across the cluster.
QAC configuration is via the sysctl ‘efs.quota.reorganize.qac_ratio’, which increases the number of accounting constituents, which are in turn spread across a much larger number of nodes and drives. This provides better scalability by increasing aggregate throughput and reduces latencies on heavy create/delete activities when quotas are configured.
Using this parameter, the internally calculated QAC count for each quota is multiplied by the specified value. If a workflow experiences write performance issues, and it has many writes to files or directories governed by a single quota, then increasing the QAC ratio may significantly improve write performance.
The qac_ratio can be reconfigured to from its default value of none up to the maximum value of 8 via the following CLI command:
# isi_sysctl_cluster efs.quota.reorganize.qac_ratio=8
To verify the persistent change, run:
# cat /etc/mcp/override/sysctl.conf | grep qac_ratio
Although increasing the QAC count via this sysctl can improve performance on write heavy quota domains, some amount of experimentation may be required until the ideal QAC ratio value is found.
Adjusting the ‘qac_ratio’ sysctl parameter can adversely affect write performance if you apply a value that is too high, or if you apply the parameter in an environment that does not have diminished write performance due to quota contention.
To help assess write performance while tuning the QAC ration, write latency (TimeAvg) for the NFSv3 protocol, for example, can continuously be monitored by running the following CLI command:
# isi statistics protocol --protocols nfs3 --classes write --output TimeAvg --format top