This lab will continue to talk about HDFS, one of the major Hadoop components. Learning this lab will help you understand the working principles and basic operations of HDFS, as well as access methods for WebHDFS in the Hadoop software architecture.
As its name shows, HDFS (Hadoop Distributed File System) is a component of distributed storage within the Hadoop framework, and it's fault tolerant and scalable.
HDFS can be used as part of a Hadoop cluster or as a stand-alone, universal distributed file system. For example, HBase is built on the basis of HDFS and Spark can also use HDFS as one of the data sources. Learning the architecture and basic operations of HDFS will be of great help for configuration, improvement and diagnostics of particular clusters.
HDFS is the distributed storage used by Hadoop applications, the source of data and the destination of data. HDFS clusters are mainly composed of NameNodes that manage file system metadata and DataNodes that store actual data. The architecture is shown in the following picture which depicts the interaction patterns among NameNodes, DataNodes and Clients:
This picture is quoted from Hadoop's official website.
Summary of the Introduction to HDFS:
Understanding these key points about HDFS will help in configuring, optimizing, and diagnosing Hadoop clusters effectively.
At the beginning of development, HDFS was designed to store very large files in a large cluster in a cross-node, highly reliable way. As mentioned earlier, HDFS stores files in blocks. Specifically, it stores each file as a sequence of blocks. Except for the last block, all the blocks in the file are the same size.
This picture is quoted from Hadoop's official website.
The HDFS Data Replication and High Availability:
The following is some material for extended reading:
Before writing the task code, you should first switch to the hadoop user. Double-click to open the Xfce terminal on your desktop and enter the following command. The password of the hadoop user is hadoop; it will be needed when switching users:
su - hadooplabex:~/ $ su - hadoop
hadoop:~$tip: the password of the hadoop user is hadoop
The Namenode should be initialized before using HDFS for the first time. This operation can be compared to formatting a disk, so use this command with caution when storing data on HDFS.
Otherwise restart the experiment in this section. Use the "Default Environment" and initialize HDFS with the following command:
/home/hadoop/hadoop/bin/hdfs namenode -formattip: The command above will format the HDFS file system, you need delete the HDFS data directory before running the command.
So you need stop the services about Hadoop and delete Hadoop data.
stop-all.sh
rm -rf ~/hadoopdataWhen you see the following message, the initialization is complete:
2024-03-01 11:16:10,439 INFO common.Storage: Storage directory /home/hadoop/hadoopdata/hdfs/namenode has been successfully formatted.Since HDFS is a layered distributed storage system built on top of local disks, you need to import data into it before using HDFS.
The first and most convenient way to prepare some files is to use Hadoop's configuration file as an example.
First you need to start the HDFS daemon:
/home/hadoop/hadoop/sbin/start-dfs.shView the services:
hadoop:~$ jps
8341 SecondaryNameNode
7962 NameNode
8474 Jps
8107 DataNodeCreate a directory and copy the data by entering the following command in the terminal:
cd /home/hadoop
mkdir sample_data
cp -r /home/hadoop/hadoop/etc/hadoop/* sample_data/List the contents of the directory:
hadoop:~$ ls /home/hadoop/sample_data/
capacity-scheduler.xml kms-log4j.properties
configuration.xsl kms-site.xml
...Any operation on the HDFS begins with hdfs dfs and is supplemented by the corresponding operational parameters. The most commonly used parameter is put, which is used as follows, and can be entered in the terminal:
/home/hadoop/hadoop/bin/hdfs dfs -put /home/hadoop/sample_data/hadoop-policy.xml /policy.xmlList the contents of the directory:
hadoop:~$ hdfs dfs -ls /policy.xml
-rw-r--r-- 1 hadoop supergroup 11765 2024-03-01 11:37 /policy.xmlThe last /policy.xml of the command means that the file name stored in HDFS is policy.xml and the path is / (root directory). If you want to continue using the previous file name, you can specify the path / directly.
If you need to upload multiple files, you can specify the file path of the local directory continuously and end with the HDFS target storage path:
/home/hadoop/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop
/home/hadoop/hadoop/bin/hdfs dfs -put /home/hadoop/sample_data/httpfs-env.sh /home/hadoop/sample_data/mapred-env.sh /user/hadoopList the contents of the directory:
hadoop:~$ hdfs dfs -ls /user/hadoop
Found 2 items
-rw-r--r-- 1 hadoop supergroup 1484 2024-03-01 11:42 /user/hadoop/httpfs-env.sh
-rw-r--r-- 1 hadoop supergroup 1764 2024-03-01 11:42 /user/hadoop/mapred-env.shFor specifying path-related parameters, the rules are the same as those in the Linux system. You can use wildcards (such as *.sh) to simplify the operation.
Similarly you can use the -ls parameter to list the files in the specified directory:
/home/hadoop/hadoop/bin/hdfs dfs -ls /user/hadoopThe files listed here may vary depending on the experiment environment.
If you need to see the content of a file, you can use the cat parameter. The easiest thing to think about is to specify a file path directly on HDFS. If you need to compare local directories with files on top of HDFS, you can specify their paths separately. However, it should be noted that the local directory needs to start with the file:// indicator, supplemented by the file path (such as /home/hadoop/.bashrc, don't forget the / in the front). Otherwise, any path specified here will be recognized by default as the path on HDFS:
/home/hadoop/hadoop/bin/hdfs dfs -cat file:///home/hadoop/.bashrc /user/hadoop/mapred-env.shThe output is as follows:
hadoop:~$ hdfs dfs -cat file:///home/hadoop/.bashrc /user/hadoop/mapred-env.sh
## ~/.bashrc: executed by bash(1) for non-login shells.
## see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
## for examples
## If not running interactively, don't do anything
case $- in
...If you need to copy a file to another path, you can use the cp parameter:
/home/hadoop/hadoop/bin/hdfs dfs -cp /user/hadoop/mapred-env.sh /user/hadoop/copied_file.txtSimilarly, if you need to move a file, use the mv parameter. This is basically the same as the Linux file system command format:
/home/hadoop/hadoop/bin/hdfs dfs -mv /user/hadoop/mapred-env.sh /moved_file.txtUse the lsr parameter to list the contents of the current directory, including the contents of subdirectories. The output is as follows:
hdfs dfs -lsr /If you want to append some new content to a file on HDFS, you can use the appendToFile parameter. And, when specifying the local file path to be appended, you can specify multiple of them. The last parameter will be the object to be appended. The file must exist on HDFS, otherwise an error will be reported:
echo 1 >> a.txt
echo 2 >> b.txt
/home/hadoop/hadoop/bin/hdfs dfs -appendToFile a.txt b.txt /user/hadoop/mapred-env.shYou can use the tail parameter to see the contents of the file tail (the end part of the file) to confirm whether the append was successful:
/home/hadoop/hadoop/bin/hdfs dfs -tail /user/hadoop/mapred-env.shView the output of the tail command:
hadoop:~$ echo 1 >> a.txt
echo 2 >> b.txt
hdfs dfs -appendToFile a.txt b.txt /user/hadoop/mapred-env.sh
hadoop:~$ hdfs dfs -tail /user/hadoop/mapred-env.sh
1
2If you need to delete a file or a directory, use the rm parameter. This parameter can also be accompanied by -r and -f, which have the same meanings as they have for the Linux file system command rm:
/home/hadoop/hadoop/bin/hdfs dfs -rm /moved_file.txtThe content of the file moved_file.txt will be deleted, and the command will return the following output 'Deleted /moved_file.txt'
In the previous content, we have learned how to create a directory in HDFS. In fact, if you need to create multiple directories at once, you can directly specify the paths of multiple directories as parameters. The -p parameter indicates that its parent directory will be created automatically if it does not exist:
/home/hadoop/hadoop/bin/hdfs dfs -mkdir -p /user/hadoop/dir1 /user/hadoop/dir2/sub_dir1If you want to see how much space a certain file or directory occupies, you can use the du parameter:
/home/hadoop/hadoop/bin/hdfs dfs -du /user/hadoop/The output is as follows:
hadoop:~$ hdfs dfs -du /user/hadoop/
1764 1764 /user/hadoop/copied_file.txt
0 0 /user/hadoop/dir1
0 0 /user/hadoop/dir2
1484 1484 /user/hadoop/httpfs-env.sh
4 4 /user/hadoop/mapred-env.shIn the previous section, we mainly introduced the file and directory operations in HDFS. If an application such as MapReduce is calculated and the file that records the result is generated, you can use the get parameter to export it to the local directory of the Linux system.
The first path parameter here refers to the path in HDFS, and the last path refers to the path saved in the local directory:
/home/hadoop/hadoop/bin/hdfs dfs -get /user/hadoop/mapred-env.sh /home/hadoop/exported_file.txtIf the export is successful, you can find the file in your local directory:
cd ~
lsThe output is as follows:
a.txt b.txt exported_file.txt hadoop hadoopdata sample_dataEach NameNode or DataNode runs a web server internally that displays basic information such as the current state of the cluster. In the default configuration, the home page of the NameNode is http://localhost:9870/. It lists basic statistics for DataNodes and clusters.
Open a web browser and input the following in the address bar:
http://localhost:9870/You can see the number of active DataNode nodes in the current "cluster" in Summary:
The web interface can also be used to browse directories and files inside HDFS. In the menu bar on the top, click on the âBrowse the file systemâ link under âUtilitiesâ:
Now we have finished introducing some of the basic operations of WebHDFS. More instructions can be found in the documentation for WebHDFS. This lab has come to an end. As a habit, we still need to close the Hadoop cluster:
/home/hadoop/hadoop/sbin/stop-yarn.sh
/home/hadoop/hadoop/sbin/stop-dfs.shhadoop:~$ jps
11633 JpsThis lab has introduced the architecture of HDFS. Also, we have learned the basic HDFS operation commands from the command line and then turned on to the HDFS web access pattern, which will help HDFS work as a real storage service for external applications.
This lab does not list any scenario of deleting files in WebHDFS. You can check the documentation yourself. More features are hidden in the official documentation, so make sure to keep an interest in reading the documentation.
The following is the material for extended reading:
