Linux File System and Disk Management: From Basics to Advanced Techniques

Introduction

Welcome to this lab on File System and Disk Management in Linux! This lab is designed for beginners who are just starting to explore the world of Linux system administration. We'll guide you through essential commands and concepts related to managing disk space, creating virtual disks, and maintaining filesystems. By the end of this lab, you'll have hands-on experience with fundamental Linux disk management tools.

Viewing Disk Usage with df

The df (disk free) command is your go-to tool for checking disk space usage on your Linux system. Let's explore how to use it:

Open your terminal. You should be in the /home/labex/project directory. If you're not sure, you can always check your current directory with the pwd command.
Run the following command to view disk usage:
```
df
```
You'll see output similar to this:
```
Filesystem     1K-blocks      Used Available Use% Mounted on
overlay         20971520    128744  20842776   1% /
tmpfs              65536         0     65536   0% /dev
tmpfs            3995004         0   3995004   0% /sys/fs/cgroup
shm                65536         0     65536   0% /dev/shm
/dev/vdb       104806400  57754052  47052348  56% /etc/hosts
```
Don't worry if this looks confusing at first! Let's break it down:
- Filesystem: This column shows the name of the disk or partition.
- 1K-blocks: This is the total size of the filesystem in 1-kilobyte blocks.
- Used: This shows how much space is currently in use.
- Available: This shows how much free space is left.
- Use%: This shows the percentage of the filesystem that is in use.
- Mounted on: This shows where in the directory tree the filesystem is mounted.

Now, let's make this output more human-readable. Run:

df -h

The -h option stands for "human-readable". You'll see output like this:

Filesystem      Size  Used Avail Use% Mounted on
overlay          20G  126M   20G   1% /
tmpfs            64M     0   64M   0% /dev
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
shm              64M     0   64M   0% /dev/shm
/dev/vdb        100G   56G   45G  56% /etc/hosts

Much better, right? Now the sizes are in GB and MB, which is easier to understand.

If you want to check the space on a specific filesystem, you can specify it:
```
df -h /dev/vdb
```
This will show information only for the /dev/vdb filesystem.

The df command is incredibly useful for quickly checking how much disk space you have left. If you ever run into issues where your system says it's running out of space, df is often the first command you'll use to investigate.

Examining Directory Sizes with du

While df gives us an overview of disk usage, sometimes we need to dig deeper. That's where du (disk usage) comes in. It helps us understand which directories and files are taking up the most space.

Let's start by using du in its simplest form. Run:
```
du ~
```
You'll see a long list of numbers and directory names. Each number represents the size of the directory in kilobytes. This can be overwhelming, so let's make it more manageable.
For a more readable output, use the -h option:
```
du -h ~
```
The -h option, just like with df, makes the output human-readable. You'll see sizes in KB, MB, or GB as appropriate.
Often, we just want to know the total size of a directory. For this, use:
```
du -sh ~
```
Here, -s means "summarize" and ~ represents your home directory. This command will show you the total size of everything in your home directory.
To view the sizes of immediate subdirectories in your home, use:
```
du -h --max-depth=1 ~
```
This shows the size of each subdirectory one level deep. The --max-depth=1 option limits how far du will recurse into subdirectories.
Let's check the size of items in your home directory:
```
du -sh ~/*
```
This will show the size of each non-hidden file and directory directly under your home directory.
Here's a powerful command to find the largest items in your home directory:
```
du -h ~ | sort -rh | head -n 10
```
Let's break this down:
- du -h ~ lists all files and directories in your home directory with their sizes
- sort -rh sorts this list in reverse order (largest first) and in human-readable format
- head -n 10 shows only the first 10 lines of output
- | is a pipe, which passes the output of one command as input to the next
This command is a great example of how we can combine simple Linux commands to perform more complex operations.

The du command is invaluable when you're trying to free up disk space. It helps you identify which directories or files are taking up the most room, so you know where to focus your cleanup efforts.

Creating and Managing a Virtual Disk

Before we dive in, let's understand what a virtual disk is. A virtual disk is simply a file that acts like a physical disk drive. Think of it like creating a container file that the operating system can treat as if it were a real hard drive. This is similar to how virtual machines use virtual disk files to store their data.

Why would we want to do this? Virtual disks are useful for:

Testing disk operations safely without risking real hardware
Creating isolated storage spaces
Learning about disk management without needing additional physical hardware
Creating backup images of real disks

Understanding Key Concepts

Before we proceed with the hands-on part, let's understand some important concepts:

Filesystem: Think of a filesystem as the way files and folders are organized on a disk. It's like a filing system in an office - it determines how data is stored and retrieved. Common Linux filesystems include ext4 (which we'll use), XFS, and btrfs.
Mounting: Mounting is the process of making a filesystem accessible to the operating system. When you mount a filesystem, you're telling Linux "make the contents of this disk available at this specific directory." It's similar to:
- Plugging in a USB drive (the physical connection)
- Then telling the computer where to show its contents (the mount point)
Partitions: A partition is a section of a disk that's treated as a separate unit. Think of it like dividing a large hard drive into smaller, independent sections. Reasons for partitioning include:
- Separating system files from user files
- Using different filesystems for different purposes
- Making backups easier
- Limiting the impact of disk failures

Let's create and work with a virtual disk:

First, let's create a 256MB virtual disk using the dd command:
```
dd if=/dev/zero of=virtual.img bs=1M count=256
```
Let's break this command down:
- dd is a utility for copying and converting files
- if=/dev/zero means "input file is /dev/zero" (a special file that provides endless zeros)
- of=virtual.img means "output file is virtual.img" (our new virtual disk file)
- bs=1M sets the block size to 1 megabyte (how much data to copy at once)
- count=256 means copy 256 blocks (resulting in a 256MB file)
This creates an empty file filled with zeros that we'll use as our virtual disk.
Verify the file size:
```
ls -lh virtual.img
```
You should see that virtual.img is exactly 256MB.
Now, let's format this virtual disk with an ext4 filesystem:
```
sudo mkfs.ext4 virtual.img
```
What's happening here? This command:
- Creates a new ext4 filesystem inside our virtual disk file
- Sets up the basic structure needed to store files and directories
- Is similar to formatting a new USB drive before first use
The ext4 filesystem is the default for many Linux distributions because it's reliable and well-tested.
Next, we need to create a mount point. This is the directory where the contents of our virtual disk will appear:
```
sudo mkdir /mnt/virtualdisk
```
Think of a mount point as a "window" into your virtual disk. After mounting, when you look into this directory, you're actually seeing the contents of the virtual disk.
Now we can mount the virtual disk:
```
sudo mount -o loop virtual.img /mnt/virtualdisk
```
Let's break down what's happening:
- The -o loop option tells Linux to treat our file as if it were a real disk device
- virtual.img is our source (the virtual disk we created)
- /mnt/virtualdisk is where we want the contents to appear
This is similar to what happens automatically when you plug in a USB drive, except we're doing it manually with our virtual disk file.
Let's verify that the disk is mounted:
```
mount | grep virtualdisk
```
You should see a line indicating that virtual.img is mounted on /mnt/virtualdisk.
Now that it's mounted, we can use it like any other directory. Let's create a file:
```
sudo touch /mnt/virtualdisk/testfile
ls /mnt/virtualdisk
```
You should see testfile listed.
When you're done using the virtual disk, you should unmount it:
```
sudo umount /mnt/virtualdisk
```
Unmounting removes the filesystem from that directory, ensuring that the operating system finishes all pending read and write operations before detaching it. Failing to unmount properly can lead to data corruption. Although the command syntax focuses on unmounting the directory, under the hood, the operating system knows this directory corresponds to the mounted disk image.

This process of creating, formatting, and mounting a virtual disk is very similar to what happens when you plug in a new hard drive or USB stick. The main difference is that we're doing everything with a file instead of a physical device.

Mounting a filesystem means attaching it to a specified directory so that the operating system can access the data inside the filesystem. In this lab, the virtual disk image file is treated as if it were a physical disk, and mounting makes its contents available at a particular directory (e.g., /mnt/virtualdisk).

Managing Disk Partitions with fdisk

In a real system, before you can create a filesystem, you often need to create partitions. While we can't modify actual disk partitions in this virtual environment, we can explore how to use fdisk to view partition information.

First, let's view information about all disk partitions:
```
sudo fdisk -l
```
This will display information about all disk devices and their partitions. You'll see details about the disk size, number of sectors, and partition table.
Now, let's look at the partition information for our virtual disk:
```
sudo fdisk -l virtual.img
```
This will show you the partition table of the virtual disk. Since we created the filesystem directly on the disk image without partitioning, you might see a message saying that it doesn't contain a valid partition table.

In a real system, you would use fdisk interactively to create, delete, or modify partitions. Here's a brief overview of how that would work:

You'd start fdisk with sudo fdisk /dev/sdX (replace X with the appropriate letter for the disk you want to partition)
You'd use the command 'n' to create a new partition
'd' would delete a partition
't' would change a partition's system id (which indicates the partition's intended use)
'w' would write the changes and exit

Remember, modifying partitions can lead to data loss, so always be careful and backup important data before making changes to disk partitions.

Fdisk is not limited to displaying partition information. It can also create, delete, and modify disk partitions interactively. While it is an essential tool for disk partitioning, be cautious when using fdisk to alter partitions on a system holding critical data; improper changes can lead to data loss.

Summary

Congratulations! In this lab, you've learned how to:

View disk usage with df
Examine directory sizes with du
Create, format, mount, and unmount a virtual disk
View partition information with fdisk

These skills form a foundation for more advanced Linux system administration tasks. They are crucial for managing storage, troubleshooting disk space issues, and maintaining filesystem health in Linux systems.

As an extra challenge, try to write a shell script that finds the top 10 largest files or directories in your home directory and displays their sizes in human-readable format. This will combine several of the commands you've learned in this lab.

Remember, practice is key in mastering these concepts. Don't hesitate to experiment with these commands (in a safe environment) to deepen your understanding. Good luck with your continued learning journey in Linux system administration!