OneFS CloudPools Statistics Reporting

For the longest time, the statistics from CloudPools accounts and policies were recorded but were only available via internal tools. In OneFS 9.4, these metrics are now easily accessible and presented via a new CLI ‘cloud’ option, within the familiar ’isi statistics’ CLI command set. This allows cluster administrators to gain insight into cloud accounts and policies for planning or troubleshooting CloudPools related activities.

There is no setup or configuration required in order to use CloudPools statistics, and all new and existing CloudPools accounts and policies statistics will automatically be collected and reported upon upgrade to OneFS 9.4.

The syntax for the new ‘isi statistics cloud’ CLI command is as follows:

Usage:

    isi statistics cloud <action>

        [--account <str>]

        [--policy <str>]

        [{--nodes | -n} <NODE>]

        [{--degraded | -d}]

        [--nohumanize]

        [{--interval | -i} <float>]

        [{--repeat | -r} <integer>]

        [{--limit | -l} <integer>]

        [--long]

        [--output ((Timestamp|time) | (Account|acct) | (Policy|pol) |

          (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |

          (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |

          (P-ID|policy_id|id) | Node)]

        [--sort ((Timestamp|time) | (Account|acct) | (Policy|pol) |

          (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |

          (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |

          (P-ID|policy_id|id) | Node)]

        [--format (table | json | csv | list | top)]

        [{--no-header | -a}]

        [{--no-footer | -z}]

        [{--verbose | -v}]

        [{--help | -h}]


The following CloudPools metrics and infomation are gathered and reported by the ‘isi statistics cloud list’ CLI command:

Name Description
In Bytes in.
Out Bytes out.
Reads Number of Reads.
Writes Number of writes.
Deletions Number of deletions.
Timestamp Date and time.
GUID Cluster global unique identifier.
Cloud Cloud vendor.
A-Key Cloud account key.
P-ID Cloud policy identifier.
Node Node number.

Standard CLI options are available for the command, including  JSON, Table, CSV output. The comprehensive list of these includes:

Option Description
–account Identify the account to view. Specify the account name or a phrase to match. Default is ‘all’ which will select all accounts.
–policy Identify the policy to view. Specify the policy name or a phrase to match. Default is ‘none’ which will select no policies.
–nodes Specify node(s) for which statistics should be reported.
–degraded Continue to report if some nodes do not respond.
–nohumanuze Output raw numbers without conversion to units.
–interval Wait <INTERVAL> seconds before refreshing the display.
–repeat Print the requested data <REPEAT> times (-1 for infinite).
–limit Number of statistics to display.
–long Display all possible columns.
–output Output specified column(s):

·         Timestamp | (Account|acct) | (Policy|pol) |  (Cluster-GUID|guid|cluster) | In | Out | Reads | Writes |  (Deletions|deletes|del) | (Cloud|vendor) | (A-Key|account_key|key) |

(P-ID|policy_id|id) | Node)

–sort Sort data by the specified comma-separated field(s) above. Prepend ‘asc:’ or ‘desc:’ to a field to change the sort order.
–format Display statistics in table, JSON, CSV, list or top format.
   
–no-header Do not display headers in CSV or table formats.
–no-footer Do not display table summary footer information.
–verbose Display more detailed information.
–help  

In addition to sorting and filtering results, the CloudPools statistics can also be separated by node.

In its simplest form, the OneFS 9.4 ‘isi statistics cloud list’ syntax returns the following information, in this case for a three node cluster:

# isi statistics cloud list

Account Policy In    Out      Reads Writes Deletions Cloud Node

---------------------------------------------------------------

S3       0.0B  510.2KB      0      4         0  AWS   2    

S3       0.0B    0.0KB      0      0         0  AWS   1    

S3       0.0B    0.0KB      0      0         0  AWS   3    

---------------------------------------------------------------

More detailed output can be obtained by including the ‘–long’ command argument. Additionally, the ‘-r’ flag will repeat the output the number of times specified and at an interval specified by the ‘-i’ flag. For example:

# isi statistics cloud list –i 3 -–account s3 -–policy s3policy -–long –r 2

Architecturally, the new CloudPools statistics reporting infrastructure utilizes existing OneFS daemons and systems.

Under the hood, the isi_cpool_io_d service searches the CloudPools accounts and policies every three minutes and registers/unregisters them as appropriate, while new data is sent directly to the isi_cpool_sysctl service. Raw cloud metrics are passed to the isi_stats service, which generates the ordered stats keys, which are reported by the ‘isi statistics cloud list’ CLI utility and the platform API. For example:

# https://<node_ip>:8080/platform/statistics/summary/cloud

While no new log files are introduced in support of the CloudPools statistics framework, the ‘isi_gather_info’ logfile coalescing utility does now include cloud account and policy information and statistics in OneFS 9.4, which will be particularly useful for Dell Support when troubleshoot CloudPools issues

In order to access the new CloudPools statistics, all the cluster’s nodes must be running OneFS 9.4 or higher. Also, be aware of the current limitations, which include not reporting file-based statistics such as the number and size of files archived and recalled, or limiting statistics to a specific time period.

  • No statistics available before OneFS 9.4 upgrade for existing CloudPools accounts since isi statistics wasn’t tracking metrics in prior releases.
  • CloudPools statistics does not include cloud object cache stats. However, these can be displayed as follows:
# isi statistics query history --keys cluster.cloudpools.object_cache.stats

Or via the ‘isi_test_cpool_stats’ CLI command, for example:

# isi_test_cpool_stats -Q --objcache
------------------------------------------------
Object Cache Counters
------------------------------------------------
 object_cache_hits: 22
 object_cache_misses: 2
 object_cache_overwrite: 0
 object_cache_drop: 0
 header_cache_drop: 0
 data_cache_drop: 0
 data_cache_timeout: 0
 header_cache_timeout: 0
 object_cache_bypass: 0
{{ cmo_cache_hits: 0}}
{{ data_cache_hits: 22}}
{{ header_cache_hits: 0}}
{{ data_cache_range_hits: 22}}
{{ data_cache_range_misses: 0}}
------------------------------------------------

Leave a Reply

Your email address will not be published.