🚧 Optimized Data Storage in Neo-Avalon

Introduction

In the futuristic city of Neo-Avalon, a subterranean leader known as the Underground Overseer has taken it upon themselves to safeguard the city's vast digital archives. With data constantly streaming in from various sources, the Overseer faces the challenge of efficiently storing and managing this ever-growing trove of information. The city's survival hinges on the Overseer's ability to compress and optimize data storage within the Hadoop Distributed File System (HDFS), ensuring that crucial data remains accessible while minimizing storage footprint.

The Overseer's primary objective is to implement data compression techniques within HDFS, enabling the city to preserve its digital heritage while conserving valuable resources. By mastering the art of data compression, the Overseer can ensure that Neo-Avalon's digital archives remain a beacon of knowledge for generations to come.

Explore the Hadoop Distributed File System (HDFS)

In this step, you will familiarize yourself with the Hadoop Distributed File System (HDFS) and its file management capabilities. HDFS is designed to store and manage large datasets across multiple nodes in a Hadoop cluster.

First, let's examine the contents of the HDFS root directory:

hdfs dfs -ls /

This command will list the contents of the HDFS root directory. You should see output similar to the following:

Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2023-06-01 12:00 /user

The /user directory is where user data is typically stored in HDFS. Let's create a new directory for our dataset:

hdfs dfs -mkdir /home/hadoop/datasets

Now, let's upload a sample dataset to HDFS:

hdfs dfs -put /path/to/sample_data.txt /home/hadoop/datasets/

Replace /path/to/sample_data.txt with the actual path to your sample data file on the local filesystem.

To verify that the file was uploaded successfully, you can list the contents of the /home/hadoop/datasets directory:

hdfs dfs -ls /home/hadoop/datasets

You should see the sample_data.txt file listed in the output.

Compress Data in HDFS

In this step, you will learn how to compress data stored in HDFS using various compression codecs. Compressing data can significantly reduce the storage footprint and improve data transfer efficiency.

First, let's explore the available compression codecs in Hadoop:

hdfs getconf -codecs

This command will list the compression codecs supported by your Hadoop installation. Common codecs include gzip, bzip2, lz4, and snappy.

Now, let's compress the sample_data.txt file using the gzip codec:

hdfs dfs -compress gzip /home/hadoop/datasets/sample_data.txt

This command will create a compressed version of the sample_data.txt file with the extension .gz.

To verify the compression, you can list the contents of the /home/hadoop/datasets directory:

hdfs dfs -ls /home/hadoop/datasets

You should see both the original sample_data.txt file and the compressed sample_data.txt.gz file listed in the output.

You can also specify the compression codec when creating a new file in HDFS:

hdfs dfs -D "hadoop.compress.codec=gzip" -put /path/to/new_data.txt /home/hadoop/datasets/new_data.txt.gz

In this example, the -D option sets the hadoop.compress.codec property to gzip, instructing HDFS to compress the file using the gzip codec during the upload process.

To read the contents of a compressed file, you can use the hdfs dfs -text command:

hdfs dfs -text /home/hadoop/datasets/sample_data.txt.gz

This command will decompress and display the contents of the sample_data.txt.gz file in the terminal.

Compress Data at the File or Directory Level

In this step, you will learn how to compress data at the file or directory level in HDFS using the hadoop archive command.

First, let's create a new directory and populate it with some sample data:

hdfs dfs -mkdir /home/hadoop/datasets/archive_test
hdfs dfs -put /path/to/file1.txt /home/hadoop/datasets/archive_test/
hdfs dfs -put /path/to/file2.txt /home/hadoop/datasets/archive_test/

Replace /path/to/file1.txt and /path/to/file2.txt with the actual paths to your sample data files on the local filesystem.

Now, let's create an archive of the archive_test directory using the hadoop archive command:

hadoop archive -archiveName datasets.har -p /home/hadoop/datasets/archive_test/ /home/hadoop/datasets/datasets.har

This command will create a Hadoop archive file named datasets.har in the /home/hadoop/datasets directory, containing all the files and subdirectories within the /home/hadoop/datasets/archive_test directory.

You can list the contents of the datasets.har archive using the hadoop fs -lsr command:

hadoop fs -lsr /home/hadoop/datasets/datasets.har

This command will display the contents of the datasets.har archive, including the files and subdirectories it contains.

To extract files from the archive, you can use the hadoop fs -cp command:

hadoop fs -cp /home/hadoop/datasets/datasets.har/archive_test/file1.txt /home/hadoop/datasets/extracted_file.txt

This command will copy the file1.txt file from the datasets.har archive to a new file named extracted_file.txt in the /home/hadoop/datasets directory.

Summary

In this lab, you learned how to compress data stored in the Hadoop Distributed File System (HDFS) using various compression codecs and techniques. By mastering data compression, you can optimize storage utilization and improve data transfer efficiency, enabling the Underground Overseer to safeguard Neo-Avalon's digital heritage while conserving valuable resources.

Through hands-on exercises, you explored the HDFS file system, uploaded sample datasets, and applied compression techniques such as file-level compression and directory-level archiving. You also learned how to extract files from compressed archives and navigate the compressed data in HDFS.

This lab not only equipped you with practical skills in data compression but also highlighted the importance of efficient data management in a future where digital information is the lifeblood of society. By harnessing the power of Hadoop and HDFS, you can contribute to the preservation of knowledge and ensure that the digital archives of Neo-Avalon remain accessible for generations to come.