Hadoop Yarn Commands application

Introduction

In the vast and unforgiving Sahara Desert, a renowned explorer embarks on a daring expedition to uncover the secrets of an ancient civilization. With the blazing sun overhead and the shifting sands beneath their feet, they must navigate through treacherous dunes and harsh conditions to reach their goal – a hidden archaeological site rumored to contain invaluable artifacts.

Their mission is not only to unearth these ancient relics but also to study the advanced technologies used by the long-forgotten society. Luckily, the explorer has brought along a powerful tool – the Hadoop YARN (Yet Another Resource Negotiator) – a cluster management technology that will help them process and analyze the vast amounts of data they'll encounter during their quest.

Skills Graph

Explore the Hadoop YARN Resource Manager

In this step, we'll explore the Hadoop YARN Resource Manager, which is responsible for managing the cluster resources and scheduling applications.

First, let's switch to the hadoop user by running the following command in the terminal:

su - hadoop

Next, we'll retrieve information about the YARN Resource Manager by executing the following command:

yarn node --list

This command will display a list of active NodeManagers in the Hadoop cluster, along with their respective statuses, nodeIDs, and other relevant information.

Here's an example output:

2024-03-16 22:35:03,954 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
Total Nodes:1
         Node-Id	     Node-State	Node-Http-Address	Number-of-Running-Containers
iZj6cb368r3kuiwgkci85bZ:39587	        RUNNING	iZj6cb368r3kuiwgkci85bZ:8042	                           0

Submit a YARN Application

In this step, we'll learn how to submit a YARN application to the cluster. We'll use a simple case that calculates the pi.

yarn jar /home/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar pi  3 3

The output will contain the result of the calculation.

Job Finished in 19.117 seconds
Estimated value of Pi is 3.55555555555555555556

Summary

In this lab, we explored the Hadoop YARN (Yet Another Resource Negotiator) and learned how to interact with the Resource Manager to manage cluster resources and schedule applications. We started by listing the active NodeManagers in the cluster, which gave us insight into the available resources.

Next, we submitted a YARN application that computing the value of pi to 25 digits. The application was successfully submitted and executed, demonstrating the ability of YARN to efficiently allocate resources and execute distributed tasks.

Through this lab, I gained practical knowledge of YARN commands and how to leverage the power of Hadoop for distributed computing tasks. The combination of theoretical concepts and hands-on exercises solidified my understanding of this powerful technology, preparing me for real-world scenarios where efficient data processing and analysis are crucial.