Introduction
In the ancient land of Egypt, the Pharaoh's palace stood as a magnificent testament to the kingdom's power and prosperity. However, beneath the gilded surface, a crisis was brewing. The Pharaoh's vast storehouses, filled with the bounty of the Nile, were in disarray. The priests, responsible for managing the distribution of resources, struggled to keep up with the demands of the people.
Enter Amenhotep, a brilliant young priest tasked with restoring order to the kingdom's resources. His mission was to develop a system that would ensure fair and efficient allocation of the precious goods, ensuring that every citizen received their due share.
The Pharaoh, impressed by Amenhotep's intellect and dedication, granted him access to the latest technological marvel – the Hadoop YARN (Yet Another Resource Negotiator) system. With this powerful tool at his disposal, Amenhotep set out to learn the intricacies of the Applying Scheduler, a key component of YARN that would help him achieve his goal.
Understanding the Hadoop YARN Schedulers
In this step, we will explore the different scheduling policies available in Hadoop YARN and inspect the running scheduler.
First, switch to the Hadoop user. Run all commands in this lab in this terminal:
su - hadoop
Set the configuration directory explicitly. In this image Hadoop is installed in /home/hadoop/hadoop; /etc/hadoop/conf is not used.
export HADOOP_CONF_DIR=/home/hadoop/hadoop/etc/hadoop
Hadoop YARN supports the Fair Scheduler and the Capacity Scheduler. The Fair Scheduler shares resources among applications over time, with queue weights affecting their relative shares. The Capacity Scheduler allocates guaranteed capacities to queues in a hierarchy, while allowing unused resources to be shared.
Hadoop 3.3.6 does not provide a yarn scheduler subcommand. Query the ResourceManager's REST API to inspect the scheduler that is actually running, rather than assuming that a configuration file has been applied. Save its response:
curl -fsS --retry 15 --retry-connrefused --retry-delay 1 http://localhost:8088/ws/v1/cluster/scheduler -o /home/hadoop/scheduler-initial.json
Display the active scheduler type:
python3 - <<'PY'
import json
with open('/home/hadoop/scheduler-initial.json') as stream:
info = json.load(stream)['scheduler']['schedulerInfo']
print(info['type'])
PY
Expected output in the fresh lab environment:
capacityScheduler
Keep this terminal open for the next steps. We will configure each scheduler through Hadoop's configuration files and restart the ResourceManager to load the selected scheduler class.
Configuring the Fair Scheduler
In this step, we will configure the Fair Scheduler to share resources between the citizens and priests queues.
As the Hadoop user, create the allocation file by running this entire block:
cat > /home/hadoop/fair-scheduler.xml <<'XML'
<?xml version="1.0"?>
<allocations>
<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
<queue name="root">
<weight>1.0</weight>
<queue name="citizens">
<weight>1.0</weight>
<minResources>1024 mb, 1 vcores</minResources>
</queue>
<queue name="priests">
<weight>2.0</weight>
<minResources>2048 mb, 2 vcores</minResources>
</queue>
</queue>
</allocations>
XML
The citizens queue has weight 1 and a minimum of 1024 MB memory and 1 vcore. The priests queue has weight 2 and a minimum of 2048 MB memory and 2 vcores. Minimum resources are targets subject to available cluster resources; the weights influence how remaining resources are shared.
Select the Fair Scheduler and point it at the allocation file in yarn-site.xml. The following command preserves the existing NodeManager settings:
python3 - <<'PY'
import os
import xml.etree.ElementTree as ET
path = os.path.join(os.environ['HADOOP_CONF_DIR'], 'yarn-site.xml')
tree = ET.parse(path)
root = tree.getroot()
settings = {
'yarn.resourcemanager.scheduler.class':
'org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler',
'yarn.scheduler.fair.allocation.file': '/home/hadoop/fair-scheduler.xml',
}
for name, value in settings.items():
prop = next((p for p in root.findall('property') if p.findtext('name') == name), None)
if prop is None:
prop = ET.SubElement(root, 'property')
ET.SubElement(prop, 'name').text = name
ET.SubElement(prop, 'value')
prop.find('value').text = value
tree.write(path, encoding='UTF-8', xml_declaration=True)
PY
Changing the scheduler class requires a ResourceManager restart. This practice cluster has no running applications. Stop and start only the ResourceManager:
yarn --daemon stop resourcemanager
yarn --daemon start resourcemanager
Query the running scheduler again. The retry options allow time for the ResourceManager to start; brief connection errors during startup can occur.
curl -fsS --retry 15 --retry-connrefused --retry-delay 1 http://localhost:8088/ws/v1/cluster/scheduler -o /home/hadoop/scheduler-fair.json
python3 - <<'PY'
import json
with open('/home/hadoop/scheduler-fair.json') as stream:
info = json.load(stream)['scheduler']['schedulerInfo']
print(info['type'])
for queue in info['rootQueue']['childQueues']['queue']:
if queue['queueName'] in ('root.citizens', 'root.priests'):
resources = queue['minResources']
print(queue['queueName'], resources['memory'], resources['vCores'])
PY
Expected output (queue order may vary):
fairScheduler
root.citizens 1024 1
root.priests 2048 2
Complete this step's check before switching schedulers in Step 3.
Configuring the Capacity Scheduler
In this step, we will switch to the Capacity Scheduler and give each queue 50% of the cluster's capacity.
As the Hadoop user, write capacity-scheduler.xml directly into Hadoop's configuration directory. A file in /home/hadoop alone would not be loaded by this scheduler:
cat > /home/hadoop/hadoop/etc/hadoop/capacity-scheduler.xml <<'XML'
<?xml version="1.0"?>
<configuration>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>citizens,priests</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.citizens.capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.priests.capacity</name>
<value>50</value>
</property>
</configuration>
XML
The child queue capacities sum to 100%. Select the Capacity Scheduler in yarn-site.xml:
sed -i 's/org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler/org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler/' /home/hadoop/hadoop/etc/hadoop/yarn-site.xml
Restart the ResourceManager to load the new scheduler class and queue configuration:
yarn --daemon stop resourcemanager
yarn --daemon start resourcemanager
Inspect the active scheduler and its queues:
curl -fsS --retry 15 --retry-connrefused --retry-delay 1 http://localhost:8088/ws/v1/cluster/scheduler -o /home/hadoop/scheduler-capacity.json
python3 - <<'PY'
import json
with open('/home/hadoop/scheduler-capacity.json') as stream:
info = json.load(stream)['scheduler']['schedulerInfo']
print(info['type'])
for queue in info['queues']['queue']:
print(queue['queueName'], queue['capacity'])
PY
Expected output (queue order may vary):
capacityScheduler
citizens 50.0
priests 50.0
Each queue has a guaranteed capacity of 50% when both need resources. Unused capacity can be shared; these values do not force idle resources to remain unused.
Summary
In this lab, we learned how to configure and apply different scheduling policies in Hadoop YARN to manage resource allocation effectively. By mastering the Fair Scheduler and the Capacity Scheduler, Amenhotep can ensure fair distribution of resources among the kingdom's citizens and prioritize critical tasks performed by the priests.
Through this hands-on experience, we gained a deeper understanding of the powerful capabilities of Hadoop YARN and its ability to manage resources in a complex environment. By applying the knowledge gained from this lab, we can build efficient and fair resource management systems tailored to the specific needs of our organization.
