Hadoop FS Shell copyFromLocal/get

Introduction

In a galaxy far far away, there exists a ongoing space war between different factions. The galaxy is also bustling with space traders trying to make profits amidst the chaos. Our story focuses on one such space trader who needs to transfer valuable data files using Hadoop's HDFS file system. The trader's goal is to successfully copy files from their local system to Hadoop using the copyFromLocal command and retrieve files from Hadoop to their local system using the get command.

Skills Graph

Copy File to Hadoop

In this step, the space trader needs to copy a file named data.txt from their local system to Hadoop HDFS.

Open the terminal and follow the steps below to get started.

1. Switch to the hadoop user for proper permissions:

   su - hadoop

2. Create a directory named space_data in Hadoop:

   hdfs dfs -mkdir /space_data

3. Copy the local file data.txt to Hadoop:

   hdfs dfs -copyFromLocal /home/hadoop/data.txt /space_data/

4. Verify that the file was successfully copied:

   hdfs dfs -ls /space_data

Retrieve File from Hadoop

Now, the space trader needs to retrieve the file data.txt from Hadoop and save it to their local system.

1. Change to the directory where you want to save the retrieved file:

   mkdir /home/hadoop/space_data && cd /home/hadoop/space_data

2. Retrieve the file data.txt from Hadoop to the local system:

   hdfs dfs -get /space_data/data.txt

3. Verify that the file was successfully retrieved:

   ls -l data.txt

Summary

In this lab, we simulated a space trading scenario where a trader needed to transfer data files between their local system and Hadoop HDFS. By practicing the copyFromLocal and get commands in Hadoop FS Shell, users can learn how to efficiently move files to and from Hadoop's distributed file system. This hands-on experience helps in understanding the basic file management operations in Hadoop and enhances one's skills in working with big data ecosystems.