Whispering Woods Node Manager Quest

Introduction

Deep within the enchanted Whispering Woods, a mystical realm where trees danced to the melody of the wind, there lived a wise and benevolent sorceress named Willow. Her cottage stood at the heart of the forest, a sanctuary of ancient knowledge and magic. Willow's mission was to maintain the delicate balance of the woodland realm and guide those who sought her counsel.

One day, a young apprentice named Aiden stumbled into the Whispering Woods, seeking wisdom and guidance. Aiden had heard tales of Willow's mastery over the powerful Hadoop cluster, a system that could process vast amounts of data with unparalleled efficiency. Determined to learn the ways of this remarkable technology, Aiden sought out Willow's cottage, hoping to become her student and unravel the secrets of Hadoop's Node Manager.

Skills Graph

Explore the Node Manager's Role

In this step, you will learn about the role of the Node Manager in the Hadoop YARN architecture.

The Node Manager is a vital component of the Hadoop YARN (Yet Another Resource Negotiator) framework. It is responsible for managing the resources of individual nodes within a Hadoop cluster. Each node in the cluster runs a Node Manager instance, which communicates with the Resource Manager to receive and execute tasks.

Here's how the Node Manager works:

1. Node Registration: When a Node Manager starts up, it registers itself with the Resource Manager, providing information about the available resources on its node, such as CPU, memory, and disk space.
2. Container Management: The Node Manager is responsible for creating and managing containers, which are isolated execution environments for tasks. Each container has a specific resource allocation defined by the Resource Manager.
3. Task Execution: When the Resource Manager assigns a task to a node, the Node Manager creates a container and launches the task within it. The Node Manager monitors the task's execution and reports its status back to the Resource Manager.
4. Resource Monitoring: The Node Manager continuously monitors the resource usage of each container and node, ensuring that tasks do not consume more resources than allocated.
5. Health Monitoring: The Node Manager also monitors the health of the node itself, checking for issues like disk failures or network connectivity problems. If a node becomes unhealthy, the Node Manager can report this to the Resource Manager, which can then take appropriate actions, such as restarting or rescheduling tasks.

To explore the Node Manager's role, let's first switch to the hadoop user:

su - hadoop

Next, we can check the status of the Node Manager by running the following command:

yarn node -status <Node-Id>

tips: you can find the 'Node-Id' by yarn node -list command.

This command will display information about the running Node Manager, including its address, the resources available on the node, and the currently running containers.

hadoop:~/ $ yarn node -status iZj6c4hvgdd6j6qljtbxoaZ:39885          [21:53:30]
2024-03-23 21:54:08,741 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2024-03-23 21:54:09,119 INFO conf.Configuration: resource-types.xml not found
2024-03-23 21:54:09,128 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
Node Report : 
	Node-Id : iZj6c4hvgdd6j6qljtbxoaZ:39885
	Rack : /default-rack
	Node-State : RUNNING
	Node-Http-Address : iZj6c4hvgdd6j6qljtbxoaZ:8042
	Last-Health-Update : Sat 23/Mar/24 09:52:56:762CST
...

Examine Node Manager Log Files

In this step, you will learn how to examine the log files generated by the Node Manager, which can provide valuable insights into its operations and any potential issues.

The Node Manager log files are located in the /home/hadoop/hadoop/logs directory. Here's how you can access and view these logs:

1. First, navigate to the log directory:

cd /home/hadoop/hadoop/logs

2. List the available log files:

ls

You should see files like log and out about 'nodemanager'.

1. To view the log file contents, you can use a text editor like nano or a command-line tool like tail or less. For example:

tail -n 100 hadoop-hadoop-nodemanager-iZj6c0nuyqgkz1limqj5htZ.log

This command will display the last 100 lines of the Node Manager log file.

...
2024-03-04 13:39:01,626 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as iZj6c0nuyqgkz1limqj5htZ:41069 with total resource of <memory:8192, vCores:8>
...

The log files contain various types of information, including:

• Node Manager startup and shutdown events
• Container allocations and launches
• Resource usage and monitoring data
• Error messages and warnings

By examining the log files, you can troubleshoot issues related to the Node Manager, such as failed container launches, resource contention, or node health problems.

Configure Node Manager Properties

In this step, you will learn how to configure the Node Manager's properties to customize its behavior and resource allocation.

The Node Manager properties are defined in the yarn-site.xml configuration file, typically located in the /home/hadoop/hadoop/etc/hadoop directory. Here's how you can modify these properties:

1. Navigate to the Hadoop configuration directory:

cd /home/hadoop/hadoop/etc/hadoop

2. Open the yarn-site.xml file in a text editor:

vim yarn-site.xml

3. Locate the yarn.nodemanager.resource.memory-mb property, which specifies the maximum amount of physical memory (in megabytes) that can be allocated for containers on the node. You can adjust this value based on your cluster's memory requirements.

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>8192</value>
</property>

4. Another important property is yarn.nodemanager.resource.cpu-vcores, which determines the number of CPU cores that can be allocated for containers on the node.

<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>8</value>
</property>

5. After making your changes, save the file and exit the text editor.

6. For the changes to take effect, you need to restart the Node Manager:

stop-yarn.sh
start-yarn.sh

By adjusting these properties, you can configure the Node Manager to allocate resources based on your cluster's requirements and workload characteristics.

Summary

In this lab, you explored the world of Hadoop's Node Manager, a vital component of the YARN framework. You journeyed through the enchanted Whispering Woods, guided by the wise sorceress Willow, and learned about the Node Manager's role in managing resources, executing tasks, and maintaining the health of a Hadoop cluster.

Through hands-on steps, you gained practical experience in examining the Node Manager's status, analyzing its log files, and configuring its properties to customize resource allocation. By mastering the Node Manager, you unlocked the power to efficiently process vast amounts of data within the Hadoop ecosystem.

This lab not only equipped you with technical skills but also fostered a deeper appreciation for the magical realm of data processing. Just as Willow maintained the delicate balance of the woodland realm, you now possess the knowledge to harness the power of the Node Manager and ensure the optimal performance and stability of your Hadoop cluster.