Introduction
This blog post walks you through the process of upgrading Hortonworks Data Platform (HDP) 2.6.5 to HDP 3.0.1 or HDP3.1.0 on DellEMC Isilon OneFS 8.1.2/OneFS 8.2 This is intended for systems administrators, IT program managers, IT architects, and IT managers who are upgrading Hortonworks Data Platform installed on OneFS 8.1.2.0. or later versions
There are two official ways to upgrade to HDP 3.* as follows:
-
- Deploy a fresh HDP 3.* cluster and migrate existing data using Data Lifecycle Manager or distributed copy (distcp).
- Perform an in-place upgrade of an existing HDP 2.6.x cluster.
This post will demonstrate in-place upgrades. Make sure your cluster is ready and meets all the success criteria as mentioned here and in the official Hortonworks Upgrade documentation.
The installation or upgrade process of the new HDP 3.0.1 and later versions on Isilon OneFS 8.1.2 and later versions has changed as follows:
The OneFS is not presented as a host to the HDP cluster anymore, and instead, OneFS is internally managed as a dedicated service in place of HDFS by installing a management pack called the Ambari Management Pack for Isilon OneFS. It is a software component that can be installed on the Ambari Server to define OneFS as a service in a Hadoop cluster. The management pack allows an Ambari administrator to start, stop, and configure OneFS as a HDFS storage service. This provides native NameNode and DataNode capabilities similar to traditional HDFS.
This management pack is OneFS release-independent and can be updated in between releases if needed.
Prerequisites
We will perform the upgrade in two parts: first we will make the changes on the OneFS host and followed by updates on the HDP cluster.
OneFS Host Preparation
The step-by-step process to prepare the OneFS host for the HDP upgrade is as follows:.
-
- First make sure the Isilon OneFS cluster is running 8.1.2 installed with the latest patch available. Check DellEMC support or Current Isilon OneFS Patches
- HDP 3.0.1 comes with TLSv2.0 service which relies on the yarn-ats user and a dedicated HBase storage in the back-end for Yarn apps and jobs framework metrics collections. For this, we create two new users yarn-ats and yarn-ats-hbase on the OneFS host.
Login to the Isilon OneFS terminal node using root credentials, and run the following commands:
isi auth group create yarn-ats isi auth users create yarn-ats --primary-group yarn-ats --home-directory=/ifs/home/yarn-ats isi auth group create yarn-ats-hbase isi auth users create yarn-ats-hbase --primary-group yarn-ats-hbase --home-directory=/ifs/home/yarn-ats-hbase
- Once the new users are created, you need to map yarn-ats-hbase to yarn-ats on the OneFS host. This step is required only if you are going to secure the HDP cluster with Kerberization.
isi zone modify --add-user-mapping-rules="yarn-ats-hbase=>yarn-ats" –-zone=ZONE_NAME
This user mapping depends on the mode of Timeline Service 2.0 Installation. Read those instructions carefully and opt for the deployment mode to avoid ats-hbase service failure.
You can skip the yarn-ats-hbase to yarn-ats user mapping in the following two cases:
-
- Renaming yarn-ats-hbase principals to yarn-ats during Kerberization if Timeline Service V2.0s are deployed in Embedded or System Service mode.
- There is no need to set user mapping if TLSv2.0 is configured on external Hbase.
HDP Cluster preparation and upgrade
Follow the steps as documented. The steps must to meet all of the prerequisites in the Hortonworks upgrade document.
- Before starting the process, make sure the HDP 2.6.5 cluster is healthy by doing a service check, and address all of the alerts, if any display.
- Now stop the HDFS service and all other components running on the OneFS host.
- Delete the Datanode/Namenode/SNamenode using the following curl command:
Note that before DN/NN and SNN are deleted, you’ll see something like the following:
Use the following curl commands to delete the DN, NN and SNN:
export AMBARI_SERVER=<Ambar server IP/FQDN> export CLUSTER=<HDP2.6.5 cluster name> export HOST=<OneFS host FQDN> curl -u admin:admin -H "X-Requested-By: Ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/$CLUSTER/hosts/$HOST/host_components/DATANODE curl -u admin:admin -H "X-Requested-By: Ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/$CLUSTER/hosts/$HOST/host_components/NAMENODE curl -u admin:admin -H "X-Requested-By: Ambari" -X DELETE http://$AMBARI_SERVER:8080/api/v1/clusters/$CLUSTER/hosts/$HOST/host_components/SECONDARY_NAMENODE
After the deleting DN/NN and SNN, you’ll see something similar to the following:
- Manually delete the OneFS host from the Ambari Server UI.
Following the steps from five to nine are critical and are related to the Hortonworks HDP upgrade process. Refer to the Hortonworks upgrade documentations or consult the Hortonworks support if necessary.
Note: Steps five to nine in the HDP upgrade process below are related to the services running on our POC cluster. You’ll have to do backup, migration, upgrades to the necessary service as described in the Hortonworks documentation before going to step 10.
———-
- Upgrade Ambari Server/agent to 2.7.1, by follow the Hortonworks Ambari Server upgrade document.
- Register and install HDP 3.0.1, by following the steps in this Hortonworks HDP register and install target version guide.
- Upgrade Ambari metrics, by following the steps in this upgrade ambari metrics guide
- Note: This next step is critical: Perform a service check on all the services and make sure to address all alerts if any.
- Click upgrade and complete the upgrade process. Address any issues encountered before proceeding to avoid service failures after finalizing the upgrade.
A screen similar to the following displays:
———–
After the successful upgrade to HDP 3.0.1, continue installing Ambari Management pack for Isilon OneFS on the upgraded Ambari Server.
- For the Ambari Server Management Pack installation, login to the Ambari Server terminal, download the management pack, install, and then restart the Ambari server.
a. Download the Ambari Management Pack for Isilon OneFS from here
b. Install the management pack as shown below. Once it is installed, the following displays: Ambari Server ‘install-mpack’ completed successfully.
root@RDUVNODE334518:~ # ambari-server install-mpack --mpack=isilon-onefs-mpack-0.1.0.0.tar.gz --verbose
Using python /usr/bin/python
Installing management pack
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Installing management pack isilon-onefs-mpack-0.1.0.0-SNAPSHOT.tar.gz
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Download management pack to temp location /var/lib/ambari-server/data/tmp/isilon-onefs-mpack-0.1.0.0-SNAPSHOT.tar.gz
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Expand management pack at temp location /var/lib/ambari-server/data/tmp/isilon-onefs-mpack-0.1.0.0-SNAPSHOT/
2018-11-07 06:36:39,137 - Execute[('tar', '-xf', '/var/lib/ambari-server/data/tmp/isilon-onefs-mpack-0.1.0.0-SNAPSHOT.tar.gz', '-C', '/var/lib/ambari-server/data/tmp/')] {'tries': 3, 'sudo': True, 'try_sleep': 1}
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Stage management pack onefs-ambari-mpack-0.1 to staging location /var/lib/ambari-server/resources/mpacks/onefs-ambari-mpack-0.1
INFO: Processing artifact ONEFS-addon-services of type stack-addon-service-definitions in /var/lib/ambari-server/resources/mpacks/onefs-ambari-mpack-0.1/addon-services
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
INFO: Adjusting file permissions and ownerships
INFO: about to run command: chmod -R 0755 /var/lib/ambari-server/resources/stacks
INFO:
process_pid=28352
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/stacks
INFO:
process_pid=28353
INFO: about to run command: chmod -R 0755 /var/lib/ambari-server/resources/extensions
INFO:
process_pid=28354
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/extensions
INFO:
process_pid=28355
INFO: about to run command: chmod -R 0755 /var/lib/ambari-server/resources/common-services
INFO:
process_pid=28356
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/common-services
INFO:
process_pid=28357
INFO: about to run command: chmod -R 0755 /var/lib/ambari-server/resources/mpacks
INFO:
process_pid=28358
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/mpacks
INFO:
process_pid=28359
INFO: about to run command: chmod -R 0755 /var/lib/ambari-server/resources/mpacks/cache
INFO:
process_pid=28360
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/mpacks/cache
INFO:
process_pid=28361
INFO: about to run command: chmod -R 0755 /var/lib/ambari-server/resources/dashboards
INFO:
process_pid=28362
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/dashboards
INFO:
process_pid=28363
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/stacks
INFO:
process_pid=28364
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/extensions
INFO:
process_pid=28365
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/common-services
INFO:
process_pid=28366
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/mpacks
INFO:
process_pid=28367
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/mpacks/cache
INFO:
process_pid=28368
INFO: about to run command: chown -R -L root /var/lib/ambari-server/resources/dashboards
INFO:
process_pid=28369
INFO: Management pack onefs-ambari-mpack-0.1 successfully installed! Please restart ambari-server.
INFO: Loading properties from /etc/ambari-server/conf/ambari.properties
Ambari Server 'install-mpack' completed successfully.
c. Restart the Ambari Server.
root@RDUVNODE334518:~ # ambari-server restart Using python /usr/bin/python Restarting ambari-server Waiting for server stop... Ambari Server stopped Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................ Server started listening on 8080 DB configs consistency check: no errors and warnings were found.
- Replace the HDFS service with the OneFS service; the management pack installed contains OneFS Service related settings.
For this step, delete the HDFS service, add the OneFS service installed from the Ambari Management Pack above, and copy the HDFS service configuration into the OneFS service.
a. To delete HDFS, add the OneFS service, and copy the configuration, you have an automation tool “hdfs_to_onefs_convertor.py”.
Login to the Ambari Server terminal and download the script from here.
wget --no-check-certificate https://raw.githubusercontent.com/apache/ambari/trunk/contrib/management-packs/isilon-onefs-mpack/src/main/tools/hdfs_to_onefs_convert.py
b. Now run the script by issuing the Ambari server and cluster name as the parameters. Once it completes, you see all the services up and running.
root@RDUVNODE334518:~ # python hdfs_to_onefs_convertor.py -o 'RDUVNODE334518.west.isilon.com' -c 'hdpupgd' This script will replace the HDFS service to ONEFS The following prerequisites are required: * ONEFS management package must be installed * Ambari must be upgraded to >=v2.7.0 * Stack must be upgraded to HDP-3.0 * Is highly recommended to backup ambari database before you proceed. Checking Cluster: hdpupgd (http://RDUVNODE334518.west.isilon.com:8080/api/v1/clusters/hdpupgd) Found stack HDP-3.0 Please, confirm you have made backup of the Ambari db [y/n] (n)? y Collecting hosts with HDFS_CLIENT Found hosts [u'rduvnode334518.west.isilon.com'] Stopping all services.. Downloading core-site.. Downloading hdfs-site.. Downloading hadoop-env.. Deleting HDFS.. Adding ONEFS.. Adding ONEFS config.. Adding core-site Adding hdfs-site Adding hadoop-env-site Adding ONEFS_CLIENT to hosts: [u'rduvnode334518.west.isilon.com'] Starting all services.. root@RDUVNODE334518:~ #
- At this point, you have successfully upgraded to HDP 3.0.1 and replaced the HDFS service with the OneFS service. From now on, Isilon OneFS only acts as an HDFS storage layer, so you can remove the Ambari Server and ODP Version settings from the Isilon’s HDFS settings as follows:
kbhusan-y93o5ew-1# isi hdfs settings modify --zone=System --odp-version= kbhusan-y93o5ew-1# isi hdfs settings modify --zone=System --ambari-server= kbhusan-y93o5ew-1# isi hdfs settings view Service: Yes Default Block Size: 128M Default Checksum Type: none Authentication Mode: all Root Directory: /ifs/hdfs-root WebHDFS Enabled: Yes Ambari Server: - Ambari Namenode: kb-hdp-1.west.isilon.com Odp Version: - Data Transfer Cipher: none Ambari Metrics Collector: kb-hdp-1.west.isilon.com kbhusan-y93o5ew-1#
13. Login into the Ambari Web UI and check the OneFS service and its configuration. Perform the service check.
A screen similar to the following displays:
Review the results:
Summary
In this blog, we demonstrated how you can successfully upgrade the Apache Ambari Server/agents to 2.7.1 and Hortonworks HDP 2.6.5 to HDP 3.0.1 on DellEMC Isilon OneFS 8.1.2 installed with the latest patch available. The same steps apply to upgrading the later versions of HDP3.0.1.
We installed Ambari server Management Pack for DellEMC Isilon OneFS which replaced the HDFS service to the OneFS service. This enables Ambari administrator to start, stop, and configure OneFS as a HDFS storage service, and this also provides native NameNode and DataNode capabilities like traditional HDFS to DellEMC Isilon OneFS.