Datopia with Hadoop Yarn

Introduction

In a whimsical land called Datopia, where information flows like streams of pure knowledge, there lived a curious being named Datina. Datina's sole purpose was to understand the intricate workings of the data realm and harness its potential. One day, Datina stumbled upon a mysterious force known as Hadoop, a powerful tool capable of processing vast amounts of data. However, to fully unleash its capabilities, Datina needed to master the art of executing Yarn commands and manipulating jar files.

The goal of this lab is to guide Datina through the process of utilizing Yarn commands and jar files within the Hadoop ecosystem. By completing this lab, Datina will gain the skills necessary to manage and execute applications efficiently, unlocking the true potential of Hadoop in the land of Datopia.

Skills Graph

Exploring the Hadoop Environment

In this step, we will familiarize ourselves with the Hadoop environment and ensure that all necessary components are properly configured.

First, to switch to the hadoop user:

su - hadoop

Then, verifing the Hadoop version:

hadoop version

You should see output similar to the following:

Hadoop 3.3.6
...

Listing Available Jar Files

In this step, we will learn how to list the available jar files in the Hadoop environment. These jar files contain pre-built applications and utilities that can be executed using Yarn commands.

ls $HADOOP_HOME/share/hadoop/mapreduce/*.jar

The output will display a list of jar files located in the $HADOOP_HOME/share/hadoop/mapreduce directory. These jar files can be used with Yarn commands to run various applications and utilities.

/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.3.6.jar
/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.3.6.jar
/home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.3.6.jar
...

Running a Jar File with Yarn

In this step, we will learn how to run a jar file using the yarn jar command. We will use the hadoop-mapreduce-examples jar file as an example.

Now, run the WordCount example from the hadoop-mapreduce-examples jar

yarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar wordcount /home/hadoop/input /home/hadoop/output

tip: You can read the content of the file by hadoop fs -cat /home/hadoop/input/*

The content of input file:

hello world
hello labex
hello Hadoop
hello Java

In the above command, we specify the jar file hadoop-mapreduce-examples-3.3.6.jar, the application to run wordcount, and the input and output paths /home/hadoop/input and /home/hadoop/output, respectively.

After running the command, you should see the output similar to the following:

hadoop:~/ $ hadoop fs -cat /home/hadoop/output/*                     [19:54:17]
Hadoop	1
Java	1
hello	4
labex	1
world	1

Monitoring Yarn Applications

In this step, we will learn how to monitor and manage Yarn applications using various commands.

List running Yarn applications:

yarn application -list

The exapmle output in the terminal:

UBMITTED, ACCEPTED, RUNNING, FINISHED, FAILED, KILLED] and tags: []):1
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
application_1711070937750_0001	          word count	           MAPREDUCE	    hadoop	   default	          FINISHED	         SUCCEEDED	           100%	http://iZj6cdxwclh8pms0k1vyyhZ:19888/jobhistory/job/job_1711070937750_0001

Get application status

yarn application -status <application_id>

The example output in the terminal:

hadoop:~/ $ yarn application -status application_1711070937750_0001   [9:31:46]
2024-03-22 09:33:12,186 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2024-03-22 09:33:12,521 INFO conf.Configuration: resource-types.xml not found
2024-03-22 09:33:12,522 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
Application Report :
	Application-Id : application_1711070937750_0001
	Application-Name : word count
	Application-Type : MAPREDUCE
	User : hadoop
	Queue : default
	Application Priority : 0
	Start-Time : 1711071042168
	Finish-Time : 1711071057334

Kill a running application

yarn application -kill <application_id>

The example output in the terminal:

hadoop:~/ $ yarn application -kill application_1711070937750_0001     [9:33:14]
2024-03-22 09:34:45,075 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
Application application_1711070937750_0001 has already finished

tips: you can list all applications by yarn application -list -appStates ALL

The yarn application command allows you to list, monitor, and manage Yarn applications. You can retrieve the application status, and even kill a running application using the respective subcommands.

Summary

In this lab, we embarked on a journey through the land of Datopia, where Datina, a curious being, sought to unlock the true potential of Hadoop's Yarn commands and jar files. By completing this lab, Datina gained valuable skills in listing available jar files, running applications using the yarn jar command, and monitoring and managing Yarn applications.

Through hands-on exercises, Datina learned to navigate the Hadoop environment, execute pre-built applications like WordCount, and monitor running applications using various Yarn commands. These skills not only empower Datina to harness the power of Hadoop but also lay the foundation for further exploration and mastery of the data realm.

This lab challenged me to create an engaging and informative learning experience, blending technical concepts with a whimsical narrative. By designing a fictional world and a relatable character, I aimed to make the learning process more enjoyable and accessible, especially for beginners. Additionally, I focused on providing clear instructions, example code snippets, and detailed explanations to ensure a smooth learning journey.