Introduction
You can upload a file to HDFS, but where did your terminal actually send it? A retailer's daily file might exist on a laptop, on the current server, or in shared storage. Reading the wrong copy can produce a perfectly valid calculation over the wrong data. Before managing more files, you need to recognize the storage your commands use.
Hadoop separates responsibilities. Hadoop Common supplies shared libraries and utilities. HDFS stores files. MapReduce describes and executes batch-processing tasks. YARN allocates resources for applications, including MapReduce jobs. You will use these names gradually; today you will inspect storage and configuration, without running a calculation or installing services.
This lab has three outcomes: discover the default filesystem, prove that identical path text can refer to different files, and identify the running HDFS services. You will finish with observations captured from your own environment. All fixtures are prepared independently of the previous lab. Use the Terminal; no programming is required.
Discover the Configured Destination
In this step, you will find the default filesystem and read the configured MapReduce execution choice.
Your terminal runs a client command. A client asks a service to perform work; the service can be on this machine or another machine. Configuration connects the client's short command to the right destination. Start in your local workspace:
cd /home/labex/project
The hdfs getconf command reads Hadoop configuration. Its -confKey option requests one named setting. fs.defaultFS means the default filesystem used when a Hadoop filesystem path has no explicit scheme:
hdfs getconf -confKey fs.defaultFS
Expected value:
hdfs://localhost:9000
This is a URI, an address that includes how to access a resource. Here, hdfs selects HDFS, localhost is this machine, and 9000 is the service's connection port. A port identifies a network endpoint used by a service. You do not need to open that port yourself.
Save the observed value. As in the first lab, > writes command output into a local file, replacing its previous contents:
hdfs getconf -confKey fs.defaultFS > /home/labex/project/output/default-filesystem.txt
Hadoop's configuration files use XML: a property has a name and a value. You only need to read one property now. grep selects matching text lines, and -A 1 includes the line immediately after a match:
grep -A 1 '<name>mapreduce.framework.name</name>' /etc/labex-bigdata/hadoop/mapred-site.xml
The two relevant lines are:
<name>mapreduce.framework.name</name>
<value>yarn</value>
yarn means MapReduce applications are configured to use YARN for resource management. This setting alone does not prove an application is running. HDFS is started for this lab; compute services will be used when you learn job submission.
Save those configuration lines as evidence rather than typing the expected value yourself:
grep -A 1 '<name>mapreduce.framework.name</name>' /etc/labex-bigdata/hadoop/mapred-site.xml > /home/labex/project/output/execution-setting.txt
For a tiny temporary table, your local filesystem may be enough. For archives shared by Hadoop jobs, the HDFS destination matters. Choosing a tool starts with a need, rather than uploading every file to Hadoop automatically.
Resolve the Same Path in Two Filesystems
In this step, you will read two prepared files with identical path text and prove which filesystem each command selects.
For this comparison only, setup has placed a different file at /home/labex/project/data/location.txt in each filesystem. That is possible because the local filesystem and HDFS have separate directory trees. The usual HDFS course workspace is /user/labex; today's matching path makes the distinction easy to see.
A file:/// URI explicitly selects the local filesystem. The three slashes combine the scheme separator with an absolute path. Before running the next command, predict whether it will read the local or shared version:
hdfs dfs -cat file:///home/labex/project/data/location.txt
Expected output:
Local store: draft product file
Even though you used the Hadoop client, the URI selected a local file. Now explicitly select HDFS with the service address you observed in Step 1:
hdfs dfs -cat hdfs://localhost:9000/home/labex/project/data/location.txt
Expected output:
HDFS store: shared product file
Finally, omit the scheme and service address. Predict which version the default from Step 1 will choose:
hdfs dfs -cat /home/labex/project/data/location.txt
The shared HDFS version appears. An absolute path fixes the location within a filesystem; it does not by itself choose that filesystem. Regular Linux cat /home/labex/project/data/location.txt would still read the local version.
Capture the three results independently so you can compare your predictions with evidence:
hdfs dfs -cat file:///home/labex/project/data/location.txt > /home/labex/project/output/local-result.txt
hdfs dfs -cat hdfs://localhost:9000/home/labex/project/data/location.txt > /home/labex/project/output/hdfs-result.txt
hdfs dfs -cat /home/labex/project/data/location.txt > /home/labex/project/output/default-result.txt
The HDFS and default results should match. Use diff, which prints nothing when the files match:
diff /home/labex/project/output/hdfs-result.txt /home/labex/project/output/default-result.txt
This experiment explains why a path alone can be misleading during troubleshooting: check both the path and the filesystem it belongs to.
Identify the Storage Services
In this step, you will connect the file operations to NameNode and DataNode roles and assemble your environment observations.
HDFS separates a file's name and location information from the file's data. The NameNode manages the directory tree and knows which DataNodes hold the file's blocks. DataNodes store those blocks and serve file content. Think of a library catalog and its shelves: the catalog helps locate a book; the shelves hold its pages.
HDFS client → NameNode: ask where the file's data is
HDFS client ↔ DataNode: transfer the file's data
A block is a portion of a file managed by HDFS. Later you will inspect blocks and replicas directly. For now, check that the services for these two roles are active. systemctl is-active asks the operating system for each named service's state; this command only observes state:
systemctl is-active labex-hdfs-namenode labex-hdfs-datanode
You should see two active lines, one for each service. Save that observation:
systemctl is-active labex-hdfs-namenode labex-hdfs-datanode > /home/labex/project/output/storage-services.txt
HDFS also reports its storage-node view. dfsadmin -report is a read-only administrative report. The following command uses the prepared hdfs service identity through sudo -u hdfs solely to read that report; keep normal file work under your own labex identity:
sudo -u hdfs /usr/local/bin/hdfs dfsadmin -report
Look for Live datanodes (1) and a node whose Decommission Status is Normal. Capacity, used space, host identifiers, and timestamps vary; those values do not need to match another learner's VM. A live node is communicating with the NameNode, while Normal means it is not being removed from service.
Save the report:
sudo -u hdfs /usr/local/bin/hdfs dfsadmin -report > /home/labex/project/output/storage-report.txt
This is single-node pseudo-distributed mode: separate Hadoop service processes run on one machine. In local mode, a program can operate on local files without HDFS services; in a multi-machine cluster, services and storage are spread across hosts. Seeing one DataNode here does not demonstrate surviving a whole-machine failure.
When you later submit a job, YARN's ResourceManager will allocate resources and a NodeManager will manage work on each worker machine. Today's storage observations do not require a running job.
Build one small report by concatenating your saved observations. cat accepts several local input paths in order; > saves their combined contents:
cat /home/labex/project/output/default-filesystem.txt /home/labex/project/output/execution-setting.txt /home/labex/project/output/storage-services.txt /home/labex/project/output/storage-report.txt > /home/labex/project/output/environment-report.txt
Read it back:
cat /home/labex/project/output/environment-report.txt
Locate the filesystem URI, yarn setting, two service states, and live-node count. You now have evidence connecting your commands to the actual environment. Next, you will manage a file's lifecycle within the correct filesystem.
Summary
You inspected the configured filesystem and execution choice, then used local, explicit HDFS, and default paths to prove where data came from. You connected those results to the NameNode's metadata role and the DataNode's storage role, and recognized the limits of a single-machine learning environment.
Your next task is to manage inputs deliberately: copy, archive, export, measure, and remove only the intended temporary files.
