SmartConnect, OneFS’ front-end load balancer and connection broker, is a bastion of the cluster that is intrinsically linked into many different areas of the product. Prior to OneFS 9.4, investigating the underlying cause of SmartConnect node resolution and/or connectivity issues typically required a variety of CLI tools and varied output, often making troubleshooting an unnecessarily cumbersome process.
With the addition of SmartConnect diagnostics, the initial troubleshooting process in OneFS 9.4 is now distilled down into a single ‘isi network pool status’ CLI command.
| Issue | OneFS 9.3 and Earlier Commands | OneFS 9.4 Command |
| Node down | isi stat or sysctl efs.gmp.group | isi network pool status |
| Node smart-failing | isi stat or sysctl efs.gmp.group | isi network pool status |
| Node shutdown read-only | isi stat or sysctl efs.gmp.group | isi network pool status |
| Node rebooting | sysctl efs.gmp.group | isi network pool status |
| Node draining | sysctl efs.gmp.group | isi network pool status |
| Required protocols not running | sysctl efs.gmp.group | isi network pool status |
| Node suspended in network pool | isi network pool view | isi network pool status |
| Node interfaces down | isi network interfaces list | isi network pool status |
| Node missing IPs | isi network interfaces list | isi network pool status |
| Node IPs about to move | No easy way to check | isi network pool status |
As the table above shows, this new diagnostic information source provides a single interface to view the primary causes affecting a node’s connectivity. For any given network pool, a summary of nodes and how many of them are resolvable is returned. Additionally, for each non-resolvable node, the detailed status is reported so the root cause can be pinpointed, providing enough context to narrow the scope of investigation to the correct component(s).
Specifically, the ‘isi network pool status’ CLI command in 9.4 now reports on the following attributes, making SmartConnect considerably easier to troubleshoot:
| Attribute | Possible Values |
| SC DNS Resolvable | · True
· False |
| Node State | · Up
· Draining · Smartfailing · Shutting Down · Down |
| IP Status | · Has usable IPs
· Does not have usable IPs · Does not have any configured IPs |
| Interface Status | · x/y interfaces usable |
| Protocols Running | · True
· False |
| Suspended | · True
· False |
Note that, while no additional configuration is needed, the ISI_PRIV_NETWORK privileges are required on a cluster account in order to run this CLI command.
In the following example, the CLI output clearly indicates that node 3 is down and requires attention. As such, it has no usable interfaces or IPs, no protocols running, and is not resolvable via SmartConnect DNS:
# isi network pool status subnet0.pool0 --show-all Pool ID: groupnet0.subnet0.pool0 SmartConnect DNS Overview: Resolvable: 2/3 nodes resolvable Needing Attention: 1/3 nodes need attention SC Subnet: groupnet0.subnet0 Nodes: LNN: 1 SC DNS Resolvable: True Node State: Up IP Status: Has usable IPs Interface Status: 1/1 interfaces usable Protocols Running: True Suspended: False ----------------------------------------------------------------------- LNN: 2 SC DNS Resolvable: True Node State: Up IP Status: Has usable IPs Interface Status: 1/1 interfaces usable Protocols Running: True Suspended: False ----------------------------------------------------------------------- LNN: 3 SC DNS Resolvable: False Node State: Down IP Status: Doesn't have any usable IPs Interface Status: 0/1 interfaces usable Protocols Running: False Suspended: False -----------------------------------------------------------------------
While currently, this new diagnostic information in OneFS 9.4 is only available via the CLI, it will be added to the WebUI in a future release..