Introduction
The retailer's product files are organized and accessible. Now you can answer the question from the first lab: which product labels appear most often? You could count a few lines by hand, but a growing collection needs a repeatable calculation.
MapReduce is a way to organize a batch calculation. Map turns input records into key/value pairs; Shuffle brings values with the same key together; Reduce combines each key's values into a result. For WordCount, the key is a product label and each appearance contributes the value 1. Counting words is a small example used to make the data flow visible, rather than a complete sales analytics system.
You will inspect these stages with a prepared teaching tool, then run a real, precompiled Hadoop WordCount program over the same input. The teaching tool prints algorithm stages in one process; it is not a distributed execution trace. The Hadoop run uses local mode so you can focus on the calculation before moving the same algorithm to YARN in the next lab. No Java or Python programming is required.
Turn Product Labels into Key/Value Pairs
In this step, you will inspect the input and follow each product appearance through Map.
Start in the local workspace:
cd /home/labex/project
Read the six input lines:
cat /home/labex/project/data/product-words.txt
Expected text:
notebook pen
notebook folder
pen pen
folder notebook
notebook
folder pen
Pause and count one product by hand. notebook appears on four lines, once on each; pen appears four times in total, including twice on the third line. folder appears three times. The program should produce those same totals.
A key/value pair associates a label with a value. In this algorithm, Map emits (product, 1) for every appearance. Two occurrences of pen create two separate pairs, rather than one pair per input line.
The provided wordcount-stages executable accepts a stage name and a local input file. Use its map view:
/home/labex/project/tools/wordcount-stages map /home/labex/project/data/product-words.txt
The output starts with:
notebook 1
pen 1
notebook 1
folder 1
The gap between each key and value is a tab. There are eleven pairs, one for each product appearance. This tool makes the algorithm visible; it does not start Hadoop services or schedule tasks. Capture the stage output:
/home/labex/project/tools/wordcount-stages map /home/labex/project/data/product-words.txt > /home/labex/project/output/map-pairs.txt
You have changed the representation of the input: whole lines became contributions that can be grouped by product.
Group and Reduce the Contributions
In this step, you will follow the pairs through Shuffle and Reduce, then connect the output to your manual prediction.
Map emits separate contributions. To total a product, its contributions must meet. Shuffle and sort organize pairs by key so a reducer can receive the values for a particular product. In a distributed job this can involve transferring data between tasks; the teaching tool only displays the grouped values.
Inspect the grouped view:
/home/labex/project/tools/wordcount-stages shuffle /home/labex/project/data/product-words.txt
Expected grouped values:
folder 1,1,1
notebook 1,1,1,1
pen 1,1,1,1
The keys are sorted. Each comma-separated 1 still represents an original appearance, not a new record. Save the grouping:
/home/labex/project/tools/wordcount-stages shuffle /home/labex/project/data/product-words.txt > /home/labex/project/output/shuffled-values.txt
Reduce combines the values for each key. Here it adds the integers. A different application could combine values differently; MapReduce is not limited to counting.
Inspect the reduced view:
/home/labex/project/tools/wordcount-stages reduce /home/labex/project/data/product-words.txt
Expected results:
folder 3
notebook 4
pen 4
Compare these values with the hand counts from Step 1. Save the result so you can compare it with the real Hadoop application next:
/home/labex/project/tools/wordcount-stages reduce /home/labex/project/data/product-words.txt > /home/labex/project/output/teaching-counts.txt
The result depends on all appearances of each key, even when the original appearances were on different lines. That is why the grouping stage sits between mapping and reducing.
Run the Real Hadoop WordCount Program
In this step, you will execute the same counting task with a precompiled Hadoop example in local mode.
A JAR packages Java program code. Hadoop and its example JAR are already installed, so you do not need to compile code. hadoop jar runs the named JAR; wordcount selects its WordCount example. The two -D settings below select the local execution framework and local filesystem for this invocation. The final two arguments are the input file and a new output directory:
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount -Dmapreduce.framework.name=local -Dfs.defaultFS=file:/// /home/labex/project/data/product-words.txt /home/labex/project/output/local-wordcount
Wait for the prompt. Hadoop prints progress and diagnostic messages; paths, identifiers, and timings vary. This is an actual Hadoop program using the local runner, rather than a YARN application. The lab does not start HDFS or YARN services, and both input and output are local.
List the local output directory:
ls /home/labex/project/output/local-wordcount
You should see a _SUCCESS marker and a part-... result file. _SUCCESS indicates successful output completion, while the result file contains the data you must check. The local shell's part-* pattern selects the result files. Pass their combined contents through sort to make the comparison stable:
cat /home/labex/project/output/local-wordcount/part-* | sort > /home/labex/project/output/hadoop-counts.txt
Compare them with the teaching tool's counts:
diff /home/labex/project/output/teaching-counts.txt /home/labex/project/output/hadoop-counts.txt
No differences should appear. You have connected the explained algorithm to real program output. If this run's output directory already exists, inspect its completed files rather than deleting it blindly or assuming a repeated submission will overwrite them.
One distinction will matter later: an HDFS block stores bytes, an input split describes a unit of input work, and a task executes work. These are related but are not always one-to-one. Result files are outputs from processing tasks, not copies of the original storage blocks.
Predict and Test One Input Change
In this step, you will add one product appearance to a working copy and test which result changes.
Keep the read-only raw input unchanged. Use local cat with > to write a new, writable working copy. Both paths here are local; the output file will be the one you change:
cat /home/labex/project/data/product-words.txt > /home/labex/project/output/changed-product-words.txt
Predict the new counts if you append one notebook appearance: only its total should increase, from 4 to 5. printf prints the supplied text, \n means a newline, and >> appends the output instead of replacing existing contents:
printf 'notebook\n' >> /home/labex/project/output/changed-product-words.txt
Inspect the changed input:
cat /home/labex/project/output/changed-product-words.txt
There should be one new final line. Run the same Hadoop program with only input and output paths changed:
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount -Dmapreduce.framework.name=local -Dfs.defaultFS=file:/// /home/labex/project/output/changed-product-words.txt /home/labex/project/output/changed-wordcount
Read the actual results in sorted order:
cat /home/labex/project/output/changed-wordcount/part-* | sort
Expected results:
folder 3
notebook 5
pen 4
The new Map pair belongs to the notebook group, so only that Reduce total changes. You can now explain the result rather than just recognize a successful command. In the next lab, the algorithm and counting rules stay the same while HDFS supplies the input and YARN manages execution resources.
Summary
You traced product appearances into Map pairs, grouped values through Shuffle, and combined them through Reduce. Real Hadoop WordCount results matched your hand calculation and the teaching tool, then changed in the predicted way when one input appearance was added.
You used the local runner to isolate the algorithm. Next, you will keep that algorithm and learn how an actual YARN application reads HDFS inputs and produces HDFS results.
