For the longest time, the statistics from CloudPools accounts and policies were recorded but were only available via internal tools. In OneFS 9.4, these metrics are now easily accessible and presented via a new CLI ‘cloud’ option, within the familiar ’isi statistics’ CLI command set. This allows cluster administrators to gain insight into cloud accounts and policies for planning or troubleshooting CloudPools related activities.
There is no setup or configuration required in order to use CloudPools statistics, and all new and existing CloudPools accounts and policies statistics will automatically be collected and reported upon upgrade to OneFS 9.4.
The syntax for the new ‘isi statistics cloud’ CLI command is as follows:
Usage:
isi statistics cloud <action>
[--account <str>]
[--policy <str>]
[{--nodes | -n} <NODE>]
[{--degraded | -d}]
[--nohumanize]
[{--interval | -i} <float>]
[{--repeat | -r} <integer>]
[{--limit | -l} <integer>]
[--long]
[--output ((Timestamp|time) | (Account|acct) | (Policy|pol) |
(Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |
(Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |
(P-ID|policy_id|id) | Node)]
[--sort ((Timestamp|time) | (Account|acct) | (Policy|pol) |
(Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |
(Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |
(P-ID|policy_id|id) | Node)]
[--format (table | json | csv | list | top)]
[{--no-header | -a}]
[{--no-footer | -z}]
[{--verbose | -v}]
[{--help | -h}]
The following CloudPools metrics and infomation are gathered and reported by the ‘isi statistics cloud list’ CLI command:
| Name | Description |
| In | Bytes in. |
| Out | Bytes out. |
| Reads | Number of Reads. |
| Writes | Number of writes. |
| Deletions | Number of deletions. |
| Timestamp | Date and time. |
| GUID | Cluster global unique identifier. |
| Cloud | Cloud vendor. |
| A-Key | Cloud account key. |
| P-ID | Cloud policy identifier. |
| Node | Node number. |
Standard CLI options are available for the command, including JSON, Table, CSV output. The comprehensive list of these includes:
| Option | Description |
| –account | Identify the account to view. Specify the account name or a phrase to match. Default is ‘all’ which will select all accounts. |
| –policy | Identify the policy to view. Specify the policy name or a phrase to match. Default is ‘none’ which will select no policies. |
| –nodes | Specify node(s) for which statistics should be reported. |
| –degraded | Continue to report if some nodes do not respond. |
| –nohumanuze | Output raw numbers without conversion to units. |
| –interval | Wait <INTERVAL> seconds before refreshing the display. |
| –repeat | Print the requested data <REPEAT> times (-1 for infinite). |
| –limit | Number of statistics to display. |
| –long | Display all possible columns. |
| –output | Output specified column(s):
· Timestamp | (Account|acct) | (Policy|pol) | (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes | (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) | (P-ID|policy_id|id) | Node) |
| –sort | Sort data by the specified comma-separated field(s) above. Prepend ‘asc:’ or ‘desc:’ to a field to change the sort order. |
| –format | Display statistics in table, JSON, CSV, list or top format. |
| –no-header | Do not display headers in CSV or table formats. |
| –no-footer | Do not display table summary footer information. |
| –verbose | Display more detailed information. |
| –help |
In addition to sorting and filtering results, the CloudPools statistics can also be separated by node.
In its simplest form, the OneFS 9.4 ‘isi statistics cloud list’ syntax returns the following information, in this case for a three node cluster:
# isi statistics cloud list Account Policy In Out Reads Writes Deletions Cloud Node --------------------------------------------------------------- S3 0.0B 510.2KB 0 4 0 AWS 2 S3 0.0B 0.0KB 0 0 0 AWS 1 S3 0.0B 0.0KB 0 0 0 AWS 3 ---------------------------------------------------------------
More detailed output can be obtained by including the ‘–long’ command argument. Additionally, the ‘-r’ flag will repeat the output the number of times specified and at an interval specified by the ‘-i’ flag. For example:
# isi statistics cloud list –i 3 -–account s3 -–policy s3policy -–long –r 2
Architecturally, the new CloudPools statistics reporting infrastructure utilizes existing OneFS daemons and systems.
Under the hood, the isi_cpool_io_d service searches the CloudPools accounts and policies every three minutes and registers/unregisters them as appropriate, while new data is sent directly to the isi_cpool_sysctl service. Raw cloud metrics are passed to the isi_stats service, which generates the ordered stats keys, which are reported by the ‘isi statistics cloud list’ CLI utility and the platform API. For example:
# https://<node_ip>:8080/platform/statistics/summary/cloud
While no new log files are introduced in support of the CloudPools statistics framework, the ‘isi_gather_info’ logfile coalescing utility does now include cloud account and policy information and statistics in OneFS 9.4, which will be particularly useful for Dell Support when troubleshoot CloudPools issues
In order to access the new CloudPools statistics, all the cluster’s nodes must be running OneFS 9.4 or higher. Also, be aware of the current limitations, which include not reporting file-based statistics such as the number and size of files archived and recalled, or limiting statistics to a specific time period.
- No statistics available before OneFS 9.4 upgrade for existing CloudPools accounts since isi statistics wasn’t tracking metrics in prior releases.
- CloudPools statistics does not include cloud object cache stats. However, these can be displayed as follows:
# isi statistics query history --keys cluster.cloudpools.object_cache.stats
Or via the ‘isi_test_cpool_stats’ CLI command, for example:
# isi_test_cpool_stats -Q --objcache
------------------------------------------------
Object Cache Counters
------------------------------------------------
object_cache_hits: 22
object_cache_misses: 2
object_cache_overwrite: 0
object_cache_drop: 0
header_cache_drop: 0
data_cache_drop: 0
data_cache_timeout: 0
header_cache_timeout: 0
object_cache_bypass: 0
{{ cmo_cache_hits: 0}}
{{ data_cache_hits: 22}}
{{ header_cache_hits: 0}}
{{ data_cache_range_hits: 22}}
{{ data_cache_range_misses: 0}}
------------------------------------------------