How to recover deleted files from Trash in Hadoop HDFS?

Introduction

This tutorial will guide you through the process of recovering deleted files from the Trash directory in Hadoop's Distributed File System (HDFS). Whether you accidentally deleted an important file or need to restore data, this article will provide you with the necessary steps to retrieve your lost information and maintain the integrity of your Hadoop cluster.

Skills Graph

Introduction to Hadoop HDFS

Hadoop Distributed File System (HDFS) is the primary storage system used by the Hadoop framework for big data processing. HDFS is designed to provide reliable, scalable, and fault-tolerant storage for large datasets.

What is HDFS?

HDFS is a distributed file system that runs on commodity hardware. It is designed to provide high-throughput access to application data and is suitable for applications that have large data sets. HDFS follows the master-slave architecture, where a single NameNode manages the file system namespace and regulates access to files by clients, while multiple DataNodes store and retrieve data.

Key Features of HDFS

1. Scalability: HDFS can scale to hundreds of petabytes of storage and thousands of client nodes.
2. Fault Tolerance: HDFS provides automatic data replication and recovery, ensuring that data is not lost even in the event of hardware failures.
3. High Throughput: HDFS is optimized for high-throughput access to application data and is well-suited for large data sets.
4. Compatibility: HDFS is compatible with a wide range of applications and tools, making it a versatile storage solution for big data processing.

HDFS Architecture

The HDFS architecture consists of a NameNode and multiple DataNodes. The NameNode is responsible for managing the file system namespace, while the DataNodes store and retrieve data blocks.

HDFS Commands

HDFS provides a set of command-line tools for interacting with the file system. Some common HDFS commands include:

Trash Management in HDFS

HDFS provides a Trash feature to help users recover accidentally deleted files. When a file is deleted in HDFS, it is first moved to the Trash directory instead of being permanently removed.

Enabling Trash

The Trash feature in HDFS is disabled by default. To enable it, you need to modify the core-site.xml configuration file and set the following properties:

<property>
  <name>fs.trash.interval</name>
  <value>1440</value>
</property>
<property>
  <name>fs.trash.checkpoint.interval</name>
  <value>0</value>
</property>

The fs.trash.interval property specifies the number of minutes after which the contents of the Trash directory will be permanently deleted. The fs.trash.checkpoint.interval property sets the frequency at which the Trash directory is checkpointed.

Deleting Files and Using Trash

When a file is deleted in HDFS, it is first moved to the Trash directory. You can use the following command to delete a file and move it to Trash:

hdfs dfs -rm /path/to/file

The deleted file will now be available in the Trash directory, which is located at /user/<username>/.Trash/.

Emptying Trash

To permanently delete the contents of the Trash directory, you can use the following command:

hdfs dfs -expunge

This will remove all the files from the Trash directory, and they will no longer be recoverable.

Restoring Deleted Files from Trash

If you need to restore a file that was accidentally deleted, you can use the following command to copy the file back from the Trash directory:

hdfs dfs -mv /.Trash/Current/path/to/file /path/to/restore

This will move the file from the Trash directory back to its original location.

Recovering Deleted Files from Trash

When a file is deleted in HDFS, it is first moved to the Trash directory, where it is stored for a specified period of time before being permanently deleted. This provides a way for users to recover accidentally deleted files.

Locating Deleted Files in Trash

To locate a deleted file in the Trash directory, you can use the following command:

hdfs dfs -ls /.Trash/Current/

This will list all the files and directories that are currently in the Trash.

Restoring Deleted Files

To restore a deleted file from the Trash directory, you can use the following command:

hdfs dfs -mv /.Trash/Current/path/to/file /path/to/restore

This will move the file from the Trash directory back to its original location.

Permanent Deletion and Expunge

If you want to permanently delete the contents of the Trash directory, you can use the following command:

hdfs dfs -expunge

This will remove all the files from the Trash directory, and they will no longer be recoverable.

Configuring Trash Retention

The Trash feature in HDFS can be configured to control the retention period for deleted files. You can modify the core-site.xml configuration file and set the following properties:

<property>
  <name>fs.trash.interval</name>
  <value>1440</value>
</property>
<property>
  <name>fs.trash.checkpoint.interval</name>
  <value>0</value>
</property>

The fs.trash.interval property specifies the number of minutes after which the contents of the Trash directory will be permanently deleted. The fs.trash.checkpoint.interval property sets the frequency at which the Trash directory is checkpointed.

By understanding and utilizing the Trash feature in HDFS, you can effectively recover accidentally deleted files and maintain data integrity in your Hadoop cluster.

Summary

By following the instructions in this Hadoop tutorial, you will learn how to effectively manage the Trash directory, understand the process of recovering deleted files, and ensure the safety and reliability of your Hadoop HDFS data. This knowledge will empower you to maintain a robust and well-managed Hadoop ecosystem, enabling you to confidently handle data recovery scenarios and safeguard your valuable information.