OneFS Time Synchronization & NTP

We’re in the process of blog site migration. Currently, these articles are also being posted to the former Dell EMC Community Network Isilon blog, until the DECN site is finally decommissioned:

https://community.emc.com/community/products/isilon/blog

Thank you in advance for your patience during this transition period.

——————————————————

That said, there have been several recent inquiries from the field around cluster time synchronization recently. So it seemed like an article on managing time and date on Isilon would be useful – and timely.

So, OneFS provides a network time protocol (NTP) service to ensure that all nodes in a cluster can easily be synchronized to the same time source. This service automatically adjusts a cluster’s date and time settings to that of one or more external NTP servers.

NTP configuration on a cluster is performed by using the ‘isi ntp’ command line (CLI) utility, rather than modifying the nodes’ /etc/ntp.conf files manually. The syntax for this command is divided into two parts: servers and settings. For example:

# isi ntp settings

Description:

    View and modify cluster NTP configuration.

Required Privileges:

    ISI_PRIV_NTP

Usage:

    isi ntp settings <action>

        [--timeout <integer>]

        [{--help | -h}]

Actions:

    modify    Modify cluster NTP configuration.

    view      View cluster NTP configuration.

Options:

  Display Options:

    --timeout <integer>

        Number of seconds for a command timeout (specified as 'isi --timeout NNN

        <command>').

    --help | -h

        Display help for this command.


There is also an isi_ntp_config CLI command available in OneFS that provides a richer configuration set and combines the server and settings functionality :

Usage: isi_ntp_config COMMAND [ARGUMENTS ...]

Commands:

    help

      Print this help and exit.

    list

      List all configured info.

    add server SERVER [OPTION]

      Add SERVER to ntp.conf.  If ntp.conf is already

      configured for SERVER, the configuration will be replaced.

      You can specify any server option. See NTP.CONF(5)


    delete server SERVER

      Remove server configuration for SERVER if it exists.


    add exclude NODE [NODE...]

      Add NODE (or space separated nodes) to NTP excluded entry.

      Excluded nodes are not used for NTP communication with external

      NTP servers.


    delete exclude NODE [NODE...]

      Delete NODE (or space separated Nodes) from NTP excluded entry.


    keyfile KEYFILE_PATH

      Specify keyfile path for NTP auth. Specify "" to clear value.

      KEYFILE_PATH has to be a path under /ifs.


    chimers [COUNT | "default"]

      Display or modify the number of chimers NTP uses.

      Specify "default" to clear the value.

By default, if the cluster has more than three nodes, three of the nodes are selected as ‘chimers’. Chimers are nodes which can contact the external NTP servers. If the cluster comprises three nodes or less, only one node will be selected as a chimer. If no external NTP server is set, they will use the local clock instead. The other non-chimer nodes will use the chimer nodes as their NTP servers. The chimer nodes are selected by the lowest node number which is not excluded from chimer duty.

If a node is configured as a chimer. its /etc/ntp.conf entry will resemble:

# This node is one of the 3 chimer nodes that can contact external NTP

# servers. The non-chimer nodes will use this node as well as the other

# chimers as their NTP servers.

server time.isilon.com

# The other chimer nodes on this cluster:

server 192.168.10.150 iburst

server 192.168.10.151 iburst

# If none or bad connection to external servers this node may become

# the time server for this cluster. The system clock will be a time

# source and run at a high stratum

In addition to managing NTP servers and authentication, individual nodes can also be excluded from communicating with external NTP servers.

The local clock of the node is set as an NTP server at a high stratum level. In NTP, a server with lower stratum number is preferred, so if an external NTP server is set the system will prefer an external time server if configured. The stratum level for the chimer is determined by the chimer number. The first chimer is set to stratum 9, the second to stratum 11, and the others continue to increment the stratum number by 2. This is so the non-chimer nodes will prefer to get the time from the first chimer if available.

For a non-chimer node, its /etc/ntp.conf entry will resemble:

# This node is _not_ one of the 3 chimer nodes that can contact external

# NTP servers. These are the cluster's chimer nodes:

server 192.168.10.149 iburst true

server 192.168.10.150 iburst true

server 192.168.10.151 iburst true


When configuring NTP on a cluster, more than one NTP server can be specified to synchronize the system time from. This allows for full redundancy of ysnc targets. The cluster periodically contacts these server(s) and adjusts the time and/or date as necessary, based on the information it receives.

The ‘isi_ntp_config’ CLI command can be used to configure which NTP servers a cluster will reference. For example, the following syntax will add the server ‘time.isilon.com’:

# isi_ntp_config add server time.isilon.com

Alternatively, the NTP configuration can also be managed from the WebUI by browsing to Cluster Management > General Settings > NTP.

NTP also provides basic authentication-based security via symmetrical keys, if desired.

If no NTP servers are available, Windows Active Directory (AD) can synchronize domain members to a primary clock running on the domain controller(s). If there are no external NTP servers configured and the cluster is joined to AD, OneFS will use the Windows domain controller as the NTP time server. If the cluster and domain time become out of sync by more than 4 minutes, OneFS generates an event notification.

Be aware though, that if the cluster and Active Directory drift out of time sync by more than 5 minutes, AD authentication will cease to function.

If neither an NTP server or domain controller are available, the cluster’s time, date, and time zone can also be set manually using the ‘isi config’ CLI command.  For example:

1.     Run the ‘isi config’ command. The command-line prompt changes to indicate that you are in the isi config subsystem:

# isi config

Welcome to the Isilon IQ configuration console.

Copyright (c) 2001-2017 EMC Corporation. All Rights Reserved.

Enter 'help' to see list of available commands.

Enter 'help <command>' to see help for a specific command.

Enter 'quit' at any prompt to discard changes and exit.

        Node build: Isilon OneFS v8.2.2 B_8_2_2(RELEASE)Node serial number: JWXER170300301

>>>


2.     Specify the current date and time by running the date command. For example, the following command sets the cluster time to 9:20 AM on April 23, 2020:

>>> date 2020/04/23 09:20:00

Date is set to 2020/04/23 09:20:00

3.     The ‘help timezone’ command will list the available timezones. For example:

>>> help timezone


timezone [<timezone identifier>]


Sets the time zone on the cluster to the specified time zone.

Valid time zone identifiers are:

        Greenwich Mean Time

        Eastern Time Zone

        Central Time Zone

        Mountain Time Zone

        Pacific Time Zone

        Arizona

        Alaska

        Hawaii

        Japan

        Advanced

4.     To verify the currently configured time zone, run the ‘timezone’ command. For example:

>>> timezone

The current time zone is: Greenwich Mean Time

5.     To change the time zone, enter the timezone command followed by one of the displayed options. For example, the following command changes the time zone to Alaska:

>>> timezone Alaska

Time zone is set to Alaska

A message confirming the new time zone setting displays. If your desired time zone did not display when you ran the help timezone command, enter ‘timezone Advanced’. After a warning screen, you will proceed to a list of regions. When you select a region, a list of specific time zones for that region appears. Select the desired time zone (you may need to scroll), then enter OK or Cancel until you return to the isi config prompt.

6.     When done, run the commit command to save your changes and exit isi config.

>>> commit

Commit succeeded.

Alternatively, these time and date parameters can also be managed from the WebUI by browsing to Cluster Management > General Settings > Date and Time.

Leave a Reply

Your email address will not be published. Required fields are marked *