OneFS SmartConnect Diagnostics

SmartConnect, OneFS’ front-end load balancer and connection broker, is a bastion of the cluster that is intrinsically linked into many different areas of the product. Prior to OneFS 9.4, investigating the underlying cause of SmartConnect node resolution and/or connectivity issues typically required a variety of CLI tools and varied output, often making troubleshooting an unnecessarily cumbersome process.

With the addition of SmartConnect diagnostics, the initial troubleshooting process in OneFS 9.4 is now distilled down into a single ‘isi network pool status’ CLI command.

Issue OneFS 9.3 and Earlier Commands OneFS 9.4 Command
Node down isi stat  or  sysctl efs.gmp.group isi network pool status
Node smart-failing isi stat  or  sysctl efs.gmp.group isi network pool status
Node shutdown read-only isi stat  or  sysctl efs.gmp.group isi network pool status
Node rebooting sysctl efs.gmp.group isi network pool status
Node draining sysctl efs.gmp.group isi network pool status
Required protocols not running sysctl efs.gmp.group isi network pool status
Node suspended in network pool isi network pool view isi network pool status
Node interfaces down isi network interfaces list isi network pool status
Node missing IPs isi network interfaces list isi network pool status
Node IPs about to move No easy way to check isi network pool status

As the table above shows, this new diagnostic information source provides a single interface to view the primary causes affecting a node’s connectivity. For any given network pool, a summary of nodes and how many of them are resolvable is returned. Additionally, for each non-resolvable node, the detailed status is reported so the root cause can be pinpointed, providing enough context to narrow the scope of investigation to the correct component(s).

Specifically, the ‘isi network pool status’ CLI command in 9.4 now reports on the following attributes, making SmartConnect considerably easier to troubleshoot:

Attribute Possible Values
SC DNS Resolvable ·         True

·         False

Node State ·         Up

·         Draining

·         Smartfailing

·         Shutting Down

·         Down

IP Status ·         Has usable IPs

·         Does not have usable IPs

·         Does not have any configured IPs

Interface Status ·         x/y interfaces usable
Protocols Running ·         True

·         False

Suspended ·         True

·         False

Note that, while no additional configuration is needed, the ISI_PRIV_NETWORK privileges are required on a cluster account in order to run this CLI command.

In the following example, the CLI output clearly indicates that node 3 is down and requires attention. As such, it has no usable interfaces or IPs, no protocols running, and is not resolvable via SmartConnect DNS:

# isi network pool status subnet0.pool0 --show-all

Pool ID: groupnet0.subnet0.pool0


SmartConnect DNS Overview:

       Resolvable: 2/3 nodes resolvable

Needing Attention: 1/3 nodes need attention

        SC Subnet: groupnet0.subnet0


Nodes:

              LNN: 1

SC DNS Resolvable: True

       Node State: Up

        IP Status: Has usable IPs

 Interface Status: 1/1 interfaces usable

Protocols Running: True

        Suspended: False

-----------------------------------------------------------------------

              LNN: 2

SC DNS Resolvable: True

       Node State: Up

        IP Status: Has usable IPs

 Interface Status: 1/1 interfaces usable

Protocols Running: True

        Suspended: False

-----------------------------------------------------------------------

              LNN: 3

SC DNS Resolvable: False

       Node State: Down

        IP Status: Doesn't have any usable IPs

 Interface Status: 0/1 interfaces usable

Protocols Running: False

        Suspended: False

-----------------------------------------------------------------------

While currently, this new diagnostic information in OneFS 9.4 is only available via the CLI, it will be added to the WebUI in a future release..

Leave a Reply

Your email address will not be published.