Introduction
You have followed Map, Shuffle, and Reduce and checked a real local WordCount run. The retailer now wants the same counting rule applied to files in HDFS, using the environment's resource-management services. The business question stays the same; the storage and execution arrangement changes.
YARN manages resources for applications. Its ResourceManager coordinates resource allocation, while a NodeManager manages execution on a worker machine. Each submitted application has an ApplicationMaster to coordinate its work, and runs work in allocated containers, resource allocations with processes inside them. These YARN containers are not necessarily Docker containers.
You will submit the prepared WordCount JAR, identify its real YARN application, inspect its completed state and logs, and check the HDFS results. A second run will use a different output path so the original input and first result remain available. The lab prepares its own data and starts the services; you do not need the preceding lab's files or Java programming skills. Use the Terminal.
Submit the Familiar Algorithm to YARN
In this step, you will prepare HDFS input and submit the same WordCount algorithm using the configured YARN framework.
Start in your local workspace:
cd /home/labex/project
Read the prepared input:
cat /home/labex/project/data/product-words.txt
These are the same six lines used to explain the algorithm. The expected totals remain folder=3, notebook=4, and pen=4. Create a raw HDFS directory with a separate parent for results:
hdfs dfs -mkdir -p /user/labex/yarn-batch/raw /user/labex/yarn-batch/results
Upload the input:
hdfs dfs -put /home/labex/project/data/product-words.txt /user/labex/yarn-batch/raw/product-words.txt
You have already learned that -put keeps the local original. The HDFS raw directory is the job's input; results must stay outside it to prevent reading previous outputs as new inputs.
The example JAR and wordcount name are unchanged from the local run. This time, omit the local-mode overrides: the installed configuration uses mapreduce.framework.name=yarn and the HDFS default filesystem. Both final arguments below are therefore HDFS paths. The job creates its own run-1 output directory, which must not already exist:
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount /user/labex/yarn-batch/raw /user/labex/yarn-batch/results/run-1
Wait for the command to return. It prints a submitted application ID, followed by job progress and a successful completion message. Application and job IDs, timings, and progress ordering vary. You do not need to catch an instantaneous RUNNING state; completed applications can still be inspected.
Connect the command to the roles:
Client submits application → ResourceManager allocates resources
ApplicationMaster coordinates MapReduce work → NodeManager runs containers
Tasks read HDFS inputs → Tasks write HDFS results
This is still a single-machine learning environment. YARN's roles are real even though their processes share one machine. Next, you will ask YARN for direct evidence that the application completed successfully.
Inspect the Completed Application
In this step, you will retain the real application ID and use it to query status and aggregated logs.
YARN assigns an Application ID when it accepts an application. It is different from an HDFS path or a result filename. List all application states, including completed applications:
yarn application -list -appStates ALL
This fresh environment has one submitted application. Its type should be MAPREDUCE, its state FINISHED, and its final status SUCCEEDED. FINISHED alone does not prove success: the final status matters too.
Save the actual application ID without copying a value from an example. The following pipeline uses grep -o to print only a matching portion. In this pattern, [0-9]* matches the digit sequences in an application ID:
yarn application -list -appStates ALL | grep -o 'application_[0-9]*_[0-9]*' > /home/labex/project/output/application-id.txt
Read the captured ID:
cat /home/labex/project/output/application-id.txt
You should see one line beginning with application_. The rest is assigned by this environment; do not use another learner's ID.
To insert that saved value into a command, $(cat ...) runs cat and substitutes its output. Double quotes keep the substituted value together as one argument. Query that actual application:
yarn application -status "$(cat /home/labex/project/output/application-id.txt)"
Locate State, Final-State, and Application-Type. Expect FINISHED, SUCCEEDED, and MAPREDUCE. Save the report:
yarn application -status "$(cat /home/labex/project/output/application-id.txt)" > /home/labex/project/output/application-status.txt
YARN also collects container logs. The logs -applicationId command reads them for the selected application. Save the logs locally so you can inspect a manageable portion:
yarn logs -applicationId "$(cat /home/labex/project/output/application-id.txt)" > /home/labex/project/output/application.log
Use local tail -n 20 to display the last twenty lines:
tail -n 20 /home/labex/project/output/application.log
Exact log messages vary. Container headings and task messages connect the application to the work YARN actually ran. Logs help explain behavior; the successful state plus correct business output is stronger evidence than a single reassuring log line. In the next lab, you will use these same diagnostic tools on failures.
Validate and Rerun Without Polluting Input
In this step, you will check the result data, export it, and rerun into a second output path.
List the first result directory:
hdfs dfs -ls /user/labex/yarn-batch/results/run-1
The _SUCCESS marker indicates completed output. The part-* files hold the counts. As before, use a quoted HDFS pattern and sort the records for stable inspection:
hdfs dfs -cat '/user/labex/yarn-batch/results/run-1/part-*' | sort
Expected result:
folder 3
notebook 4
pen 4
Save a sorted local export using the read-back technique:
hdfs dfs -cat '/user/labex/yarn-batch/results/run-1/part-*' | sort > /home/labex/project/output/run-1-counts.txt
Before the next run, predict whether another execution over identical input should change the counts. It should not. A new output path keeps the first result and avoids the output-already-exists error:
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount /user/labex/yarn-batch/raw /user/labex/yarn-batch/results/run-2
After successful completion, read and save the second run's sorted counts:
hdfs dfs -cat '/user/labex/yarn-batch/results/run-2/part-*' | sort > /home/labex/project/output/run-2-counts.txt
Compare the two local exports:
diff /home/labex/project/output/run-1-counts.txt /home/labex/project/output/run-2-counts.txt
No differences should appear. The results are siblings outside the raw input directory, so rerunning did not add the earlier output to the next input batch. You have now checked an actual application, its logs, its result data, and repeatable behavior without changing the algorithm.
Summary
You moved a familiar counting algorithm from local execution to HDFS and YARN. You identified a real MapReduce application, inspected successful completion and container logs, and checked correct results from two output paths.
The algorithm did not change, while its storage and resource-management arrangement did. Next, you will learn to distinguish client-side problems from failures of an already submitted application and make the smallest appropriate repair.
