OneFS Netlogger

Among the useful data and network analysis tools that OneFS provides is the isi_netlogger utility. Netlogger captures IP traffic over a period of time for network and protocol analysis.

Under the hood, isi_netlogger is a python wrapper around the ubiquitous tcpdump utility. Netlogger can be run either from the OneFS command line or WebUI.

For example, from the WebUI, browse to Cluster management > Diagnostics:

Alternatively, from the OneFS CLI, the isi_netlogger command captures traffic on interface (‘-i’) over a timeout period of minutes (‘-t’), and stores a specified number of log files,as defined by the keep_count, or ‘-k’ parameter.

Using the ‘-b’ bpf buffer size option will temporarily change the default buffer size while netlogger is running. Netlogger’s log files are stored by default under /ifs/netlog/<node_name>.

Here’s the basic syntax of the tool:

isi_netlogger [-c launch clustered mode (run on all nodes)]

              [-n run on specified nodes (ex: -n 1,3)]

              [-d run as daemon]

              [-q quiet mode (redirect output to logs)]

              [-k keep_count of logs (default 3, to keep all logs use 0)]

              [-t timeout (default 10) ]

              [-s snaplen (default 320) ]

              [-b bpf buffer size in bytes, KB, or MB (end with 'k' or 'm') ]

              [-i interface name[,..] | all (ex: -i em0 or -i em0,em1 or -i all)]

              [-a ARP packets included. Normally filtered out ]

              [-p print out the tcpdump command ]

              [-z do not bundle capture files (default bundling is done)]

              [-- tcpdump filtering options]

The WebUI can also be used to configure the netlogger parameters under Cluster management > Diagnostics > Netlogger settings:

Be aware that ‘isi_for_array isi_netlogger’ will consume significant cluster resources. When running the tool on a production cluster, be cognizant of the effect on the system.

When the command has completed, the capture file(s) are stored under the /ifs/netlog directory.

The following command can also be used to incorporate netlogger output files into an isi_gather_info bundle:

# isi_gather_info -n [node#] -f /ifs/netlog

To capture on multiple nodes of the cluster, the netlogger command can be prefixed by the versatile isi_for_array utility:

# isi_netlogger -n 2,3 -t 5 -k 864 -s 256

The command syntax above will create five minute incremental files on nodes 2 and 3, using a snaplength of 256 bytes, which will capture the first 256 bytes of each packet. These five-minute logs will be kept for about three days and the naming convention is of the form netlog-<node_name>-<date>-<time>.pcap. For example:

# ls

/etc/netlog/lab-cluster-1/netlog-lab_cluster-1.2021-10-18_20.24.38.pcap

When using isi_netlogger, the ‘-s’ flag needs to be set appropriately based on the protocol being to capture the right amount of detail in the packet headers and/or payload. Or, if you want the entire contents of every packet, a value of zero (‘-s 0’) can be used.

The default snaplength for netlogger is to use a snaplen of 320 bytes per packet, which is usually enough for most protocols.

However, for SMB, a snaplength of 512 is sometimes required. However, depending on a node’s traffic quantity, a snaplen of 0 (eg: capture whole packet) can potentially overwhelm the nic driver.

All the output gets written to files under /ifs/netlog and the default capture time is ten minutes (‘-t 10’).

Filters can be applied to the  filter to the end to constrain traffic to/from certain hosts or protocols. For example, to limit output to traffic between client 10.10.10.1:

# isi_netlogger -t 5 -k 864 -s 256 -- host 10.10.10.1

Or to capture NFS traffic only, filter on port 2049:

# isi_netlogger -- port 2049

Or multiple ports:

# isi_netlogger -- port 2345 or port 5432

To capture from a non-standard interface can sometime require a bit of creativity:

# isi_netlogger -p -t 5 -k 864 -s 256 -a -- " -i vlan0 host 192.168.10.1"

The -p flag to print out the tcpdump command it is running. And, essentially, anything following a double dash flag ‘–‘ is passed as a normal tcpdump filter/option.

To capture across differing interface names across the cluster:

# isi_netlogger -i \`ifconfig |grep -B2 'inet <ip_addr>.' | grep flags= | awk -F: '{ print $1 }'\` -s 0 -a

Where <ip_addr> is as much of the public IP address that is common across all the nodes. For example “192.” or “192.168.”.

To stop netlogger before a command has completed, a simple ‘isi_for_array killall tcpdump’ can be used to terminate any active tcpdump/netlogger sessions across a cluster. If any processes do remain after this, these can all be killed with a command along the lines of:

# isi_for_array -s "kill -9 \`ps auxw|grep netlog|grep -v grep|awk {'print \$2'}\`;killall -9 tcpdump"

Leave a Reply

Your email address will not be published.