Hadoop FS Shell tail

Introduction

Imagine a scenario in an ancient kingdom where a fierce dragon has been causing havoc by setting fire to villages. The king seeks a solution to track the dragon's movements and stop its destructive behavior. In this lab, you will utilize the Hadoop HDFS skill "FS Shell tail" to analyze real-time data and track the dragon's whereabouts to protect the kingdom.

Skills Graph

Check Out the Dragon Trail

In this step, you will use the "tail" command in the Hadoop FS Shell to monitor changes to a specific file, simulating dragon sighting data.

Open the terminal and follow the steps below to get started.

1. Switch to the Hadoop user for proper permissions:

   su - hadoop

2. Create a text file called dragon_sightings.txt containing some initial data.

   echo "A man saw the dragon." > /home/hadoop/dragon_sightings.txt

3. Upload the file to the /home/hadoop directory:

   hdfs dfs -put /home/hadoop/dragon_sightings.txt /home/hadoop/dragon_sightings.txt

4. Check the last few lines of the file dragon_sightings.txt to see the latest sightings.

   hdfs dfs -tail /home/hadoop/dragon_sightings.txt

Continuous Monitoring

Now, let's create a script that continuously monitors the dragon sightings file for any updates using a while loop and the "tail" command.

1. Create a script file named monitor_dragon_sightings.sh

   nano /home/hadoop/monitor_dragon_sightings.sh

   Add the following content to the file:

   #!/bin/bash
   while :
   do
    hdfs dfs -tail /home/hadoop/dragon_sightings.txt
    sleep 10
   done

   Save the file and exit the editor.

2. Make the script executable and run it to start monitoring the file.

   chmod +x monitor_dragon_sightings.sh

   ./monitor_dragon_sightings.sh

Summary

In this lab, we created a scenario where we used Hadoop FS Shell to track dragon sightings in an ancient kingdom. By practicing the "tail" command, we learned how to monitor real-time data changes and set up continuous monitoring scripts. This hands-on experience helps understand how Hadoop can be used for real-time data analysis and monitoring tasks related to big data processing. Happy tracking those dragons!