Introduction
The retailer has received a product file. One colleague needs a review copy, another needs an archived version, and you need to download the reviewed file for inspection. If these actions accidentally replace or delete the original, the team loses its trusted input.
A file's lifecycle is the set of actions it goes through: arrival, use, archiving, export, and eventual removal. HDFS provides operations for each action, but their effects differ. Uploading and downloading copy between filesystems; copying and moving within HDFS stay inside HDFS.
You will organize separate raw, archive, and scratch directories, preserve the raw file, export an archive, and remove only a disposable copy. You already know HDFS paths and upload/read commands; this lab adds careful file management. The environment and data are independent of earlier labs. Use the Terminal throughout.
Preserve a Raw Input
In this step, you will upload the retailer's input into a directory reserved for original files.
A raw file is an input kept in its original form. A separate scratch directory holds temporary work, and an archive directory holds a retained copy. Directory names do not enforce these rules automatically; they make your intentions visible.
Start in the local workspace:
cd /home/labex/project
Read the input before storing it:
cat /home/labex/project/data/products.txt
It contains:
notebook pen notebook
folder pen
notebook folder
Use -mkdir -p with three directory paths. HDFS creates each named directory and any missing parents:
hdfs dfs -mkdir -p /user/labex/file-lifecycle/raw /user/labex/file-lifecycle/archive /user/labex/file-lifecycle/scratch
Upload the local input to the raw directory:
hdfs dfs -put /home/labex/project/data/products.txt /user/labex/file-lifecycle/raw/products.txt
A successful upload normally prints nothing. If you repeat it, an existing-destination error protects the stored file from an accidental overwrite. Inspect the stored input rather than deleting it to silence the error:
hdfs dfs -cat /user/labex/file-lifecycle/raw/products.txt
The text should match the local original. The original remains on the local filesystem too: -put copies, rather than moves, its local source. Your subsequent work will use copies so this raw input stays intact.
Copy and Archive Within HDFS
In this step, you will create a review copy, then move that copy to the archive without moving the raw input.
The -cp operation copies an HDFS source to an HDFS destination. Unlike -put, both paths below refer to HDFS:
hdfs dfs -cp /user/labex/file-lifecycle/raw/products.txt /user/labex/file-lifecycle/scratch/review.txt
Inspect the review copy:
hdfs dfs -cat /user/labex/file-lifecycle/scratch/review.txt
You should see the same product labels. A different name is still the same data, not a new batch of sales.
Once a review copy is ready to retain, move it with -mv. Moving within this filesystem changes its path; it does not leave the source path in place. Before running the command, predict which file will disappear and which will remain:
hdfs dfs -mv /user/labex/file-lifecycle/scratch/review.txt /user/labex/file-lifecycle/archive/products-reviewed.txt
List scratch and archive together. -ls accepts multiple paths:
hdfs dfs -ls /user/labex/file-lifecycle/scratch /user/labex/file-lifecycle/archive
The archive entry should end in /archive/products-reviewed.txt; there should be no review.txt entry in scratch. Confirm that the raw file was unaffected:
hdfs dfs -cat /user/labex/file-lifecycle/raw/products.txt
Its three lines should still match. If you return to this step after completing it, inspect the existing archive instead of rerunning the copy/move pair: -mv has already consumed the scratch source, and an existing archive should not be overwritten casually.
Export and Measure Your Files
In this step, you will download the archived file and inspect file counts and storage usage.
The -get operation copies an HDFS source into a local destination. The first path below is HDFS; the second is local:
hdfs dfs -get /user/labex/file-lifecycle/archive/products-reviewed.txt /home/labex/project/output/products-exported.txt
The prompt should return without output. For an already completed download, inspect the local destination; downloading again to an existing file can report that it exists. Use local diff to check content, as in the first lab:
diff /home/labex/project/data/products.txt /home/labex/project/output/products-exported.txt
No differences should appear. This is an exported copy: both the HDFS archive and the local original remain.
To measure directory contents, -count prints directory count, file count, content size in bytes, and the path. It includes the specified directory itself in the directory count:
hdfs dfs -count /user/labex/file-lifecycle/raw /user/labex/file-lifecycle/archive
Each row should describe one directory containing one file. Both rows should have the same content size, because the archived bytes match the raw bytes. Save the actual rows:
hdfs dfs -count /user/labex/file-lifecycle/raw /user/labex/file-lifecycle/archive > /home/labex/project/output/file-counts.txt
The -du operation summarizes file sizes. -h chooses human-readable units when useful:
hdfs dfs -du -h /user/labex/file-lifecycle/raw
The columns show the content size, space consumed with replicas, and path. This environment uses replication factor 1, so the two size columns should agree for this file. You will explore blocks and replication in a later lab; a larger second column in another environment need not mean extra business records.
Before continuing, explain the boundaries to yourself: -cp stayed inside HDFS, -mv changed an HDFS path, and -get created a local copy.
Remove Only a Disposable Copy
In this step, you will inspect the deletion policy and remove a known temporary file while keeping the input and archive.
Deletion is a different kind of operation: once a file is no longer needed, its path may be removed. HDFS can have a Trash retention policy, but recovery is not guaranteed just because you used HDFS. First read the configured Trash interval, in minutes:
hdfs getconf -confKey fs.trash.interval
The prepared environment uses 0, meaning Trash retention is disabled. Treat a deletion here as immediate removal. A positive value in another environment enables a retention interval, but is not a replacement for keeping trusted inputs.
Create a copy whose only purpose is to practice removal:
hdfs dfs -cp /user/labex/file-lifecycle/raw/products.txt /user/labex/file-lifecycle/scratch/disposable.txt
Before deleting anything, list the exact target:
hdfs dfs -ls /user/labex/file-lifecycle/scratch/disposable.txt
Check that the path ends in scratch/disposable.txt, not raw/products.txt or the archive. The -rm operation removes the specified file. You do not need recursive deletion or a wildcard:
hdfs dfs -rm /user/labex/file-lifecycle/scratch/disposable.txt
The command reports that the named file was deleted. Repeating it would report a missing file; that is a reason to inspect state rather than broaden the deletion target. List scratch to confirm it has no remaining temporary files:
hdfs dfs -ls /user/labex/file-lifecycle/scratch
An empty directory produces no file entries. Finally, inspect the two files that must remain:
hdfs dfs -ls /user/labex/file-lifecycle/raw/products.txt /user/labex/file-lifecycle/archive/products-reviewed.txt
Both paths should be listed. You have completed a lifecycle without sacrificing the original: upload, copy, archive, export, measure, and remove only temporary work.
Summary
You kept raw data intact while making a review copy, moving that copy into an archive, and exporting it to the local filesystem. You interpreted file counts and storage sizes, inspected the Trash setting, and removed only a named disposable file.
Next, you will apply these habits to daily batches containing several files. Knowing which files belong in a batch matters as much as knowing how to upload them.
