Your First Hadoop Lab

Introduction

You are helping a retailer answer a simple question: Which products appeared most often in our stores' daily records? Before you can count anything, you need to collect the files and make sure their contents are available and correct. This lab is your first step in that journey.

From Everyday Data to Big Data

A single store might keep its daily records in a small file that you can open on a laptop. Now imagine thousands of stores sending records every day, alongside website activity and warehouse updates. You have more data to keep, new data keeps arriving, and the files may have different structures.

Big data describes data workloads whose size, arrival rate, or complexity makes them difficult to handle with the tools and resources you currently use. There is no universal number of rows that turns a file into big data. The practical questions are: Can you store the data, process it in the time available, and keep working when equipment fails?

For a small sales file, a spreadsheet, a database, or a Python script may be enough. As workloads grow, one approach is to share storage and processing across several connected computers. A group of computers working together is called a cluster; sharing work across them is called distributed computing.

Where Hadoop Fits

Apache Hadoop is an open-source collection of tools for storing and processing data across a cluster. Two parts give you a useful starting picture:

A batch is a bounded collection of data processed together, such as yesterday's files. In later labs, you will run a ready-made counting program. In this lab, you will work with HDFS, the storage part: a calculation is only useful if it reads the intended data.

Hadoop is one part of the wider data-tool landscape. Later courses introduce Hive for SQL queries over files, Spark for programmable data processing, and Kafka for ongoing event delivery. You do not need those tools to begin this lab, and you do not need to memorize their names yet.

Your First Piece of the Workflow

The retailer's simplified workflow is:

Collect product files → Store them in HDFS → Process a batch → Inspect the counts
                        You start here

Your task is to upload one file, read it back, and check that its contents were preserved. Think of this as checking an ingredient before using it in a recipe: successful storage comes before a trustworthy calculation.

The lab provides Hadoop on one machine, so you can learn real HDFS commands without installing a cluster. Local files and HDFS files still have separate paths and access commands in this environment. This setup does not demonstrate storage spread across multiple machines or recovery from a machine failure; those ideas will become clearer as you explore HDFS later.

By the end of this lab, you will be able to:

• Distinguish a local file from a file stored in HDFS.
• Create an HDFS directory and upload a file.
• Read an HDFS file and compare it with the local original.
• Upload the same data under a different destination name.

Use the Terminal tab throughout this lab. You only need basic terminal skills; Java and Python programming are not required. If paths and terminal commands are new to you, start with the Linux learning path.

Upload Your First File to HDFS

In this step, you will inspect a local file, create your own HDFS directory, and upload the file into it.

Your terminal is where you type requests. The hdfs command is a client: it sends those requests to the running HDFS service, which manages the stored files. You will see both local Linux commands and HDFS commands below; the command tells you which storage you are using.

The lab has already started the HDFS services and prepared a small input file. Click the Terminal and run each command below, waiting for the prompt to return before continuing. Copy only the command, without any prompt symbols.

Use cd to move your shell to the local project directory:

cd /home/labex/project

The path starts with /, so it is an absolute path. Your local input lives in the data directory beneath this location. The regular Linux cat command displays a local file:

cat /home/labex/project/data/products.txt

Expected contents:

notebook pen notebook
folder pen
notebook folder

Each word is a product label, and each appearance represents one recorded item in this simplified sample. You can already count five appearances of notebook and pen combined by hand: three notebooks and two pens. Across many large files, a batch program could do that counting for you.

For now, keep the question in mind but focus on storing the text exactly as it is. We use three readable lines so you can recognize a correct result immediately; this sample itself does not need Hadoop.

HDFS has its own directory tree. In this environment, hdfs dfs sends filesystem commands to HDFS. Creating a local directory with Linux mkdir would not create an HDFS directory.

Create a directory beneath your HDFS home, /user/labex. The -mkdir operation creates a directory; -p also creates missing parent directories and accepts a directory that already exists:

hdfs dfs -mkdir -p /user/labex/your-first-hadoop-lab

Success normally produces no output. The prompt returns when the operation finishes. This directory is in HDFS, even though the path uses the same slash notation as a local path.

The -put operation copies a local source file to an HDFS destination. The first path below is local; the second is in HDFS:

hdfs dfs -put /home/labex/project/data/products.txt /user/labex/your-first-hadoop-lab/products.txt

A successful upload normally produces no output and leaves the local original in place. If you repeat this command after a successful upload, HDFS reports that the destination already exists. Continue by inspecting it; you do not need to delete or overwrite it.

Use -ls to list the HDFS directory:

hdfs dfs -ls /user/labex/your-first-hadoop-lab

You should see Found 1 items and a file entry ending in:

/user/labex/your-first-hadoop-lab/products.txt

The entry also includes permissions, an owner, a group, a size, and a modification time. Those details will be explored later. For this step, the destination path confirms that the file is present in HDFS. Next, you will check what it contains.

Read Back and Check the Contents

In this step, you will read the HDFS file, save its contents locally, and compare them with the original.

The retailer will eventually count labels from the stored file. If the upload used the wrong file or lost a line, the later counts could be wrong even if the program ran successfully. A directory listing proves that a file exists, but it does not prove the contents are correct.

Before reading it back, predict what you should see: should uploading the file change any product labels or remove any lines? The HDFS -cat operation reads a file from HDFS and prints its contents in your terminal:

hdfs dfs -cat /user/labex/your-first-hadoop-lab/products.txt

Expected contents:

notebook pen notebook
folder pen
notebook folder

Compare this with the local file you inspected in Step 1. The words and line breaks should match. Notice the command difference: cat reads the local file, while hdfs dfs -cat reads the HDFS file in this environment.

To keep a local copy of the read-back result, use the shell's > operator. It redirects the command's printed output into a local file, creating that file or replacing its existing contents. The prepared output directory is for your results:

hdfs dfs -cat /user/labex/your-first-hadoop-lab/products.txt > /home/labex/project/output/products-readback.txt

No product text appears in the terminal this time because it was written into the local output file. The HDFS source is unchanged. Read the saved result with the local cat command:

cat /home/labex/project/output/products-readback.txt

It should show the same three lines. You now have a local original, an HDFS file, and a local read-back file.

Use diff to compare the original and the saved result. When two files are identical, diff prints nothing and returns successfully:

diff /home/labex/project/data/products.txt /home/labex/project/output/products-readback.txt

A return to the prompt with no differences confirms that the read-back contents match the original. If differences appear, repeat the read-back command using the exact HDFS source and local output paths above, then compare again.

Practice with a New Destination Name

In this step, you will change one part of the upload command and check whether the file contents change.

Before running the next command, predict the result: if you keep the local source but change only the HDFS destination filename, will the product labels change? Will the first HDFS file still exist?

Upload the original local file again, this time using products-copy.txt as the destination name:

hdfs dfs -put /home/labex/project/data/products.txt /user/labex/your-first-hadoop-lab/products-copy.txt

As before, a successful upload normally produces no output. If you already completed this upload, an existing-destination message means you should inspect the file rather than upload it again.

List the HDFS directory to check your prediction:

hdfs dfs -ls /user/labex/your-first-hadoop-lab

You should now see Found 2 items, with entries ending in these paths:

/user/labex/your-first-hadoop-lab/products-copy.txt
/user/labex/your-first-hadoop-lab/products.txt

Both files remain present because -put copies the local source into the named destination; it does not rename the earlier HDFS file. This second file is a practice copy of the same records, rather than a new day of sales. When you learn to prepare batch inputs, you will choose files carefully so duplicate copies are not counted twice.

Read the new HDFS file:

hdfs dfs -cat /user/labex/your-first-hadoop-lab/products-copy.txt

The output should contain the same three lines of product labels. Changing a filename does not change the file's contents.

Use the same read-back technique to save the new file locally:

hdfs dfs -cat /user/labex/your-first-hadoop-lab/products-copy.txt > /home/labex/project/output/products-copy-readback.txt

Compare that saved result with the original:

diff /home/labex/project/data/products.txt /home/labex/project/output/products-copy-readback.txt

No differences should be printed. You have checked both parts of your prediction: the first HDFS file remains, and the second contains identical data.

Before moving on, identify which commands read local files and which read HDFS files. In the next lab, you will inspect the configuration that tells the HDFS commands where to connect.

Summary

You began with a retailer's need to collect and count product records. You learned why growing data workloads can call for distributed storage and processing, and where Hadoop fits: HDFS stores files, while MapReduce can process a batch.

You completed the storage part of that workflow: you uploaded a local retail text file into HDFS, read it back, and used diff to confirm that its contents were preserved. You then predicted and checked the effect of changing only the HDFS destination name.

You practiced hdfs dfs -mkdir -p, -put, -ls, and -cat, and distinguished those HDFS operations from local cat and shell output redirection. Next, you will explore how the Hadoop client finds HDFS and how storage and computation fit together.