Linux Non-interactive Downloading

Introduction

Welcome to this beginner's lab on non-interactive downloading in Linux. The ability to download files efficiently from the command line is an essential skill for any Linux user or system administrator.

In this lab, you will learn how to use the wget command, a powerful utility that allows you to download files from the internet without manual intervention. This tool is particularly useful when you need to fetch multiple files, download in the background, or automate downloading tasks in scripts.

By the end of this lab, you will understand how to use wget to download individual files, rename downloads, and download multiple files from a list - all through the command line without graphical interfaces or interactive prompts.

Skills Graph

Setting up the Environment and Basic Downloading

In this first step, we'll create a working directory and learn how to download a single file using the wget command.

Creating a Working Directory

Let's start by creating a directory where we'll store all our downloaded files. This helps keep our files organized in a single location.

Navigate to the project directory and create a new directory called download_resources:

cd ~/project
mkdir download_resources

Understanding the wget Command

The wget command is a utility for non-interactive downloading of files from the web. Its basic syntax is:

wget [options] [URL]

Your First Download

Now, let's use wget to download a file. We'll download a Python distribution package as a test file:

cd ~/project/download_resources
wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz

When you run this command, you should see output similar to this:

--2024-01-10 10:14:51--  https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
Resolving www.python.org (www.python.org)... 151.101.76.223, 2a04:4e42:12::223
Connecting to www.python.org (www.python.org)|151.101.76.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22540566 (21M) [application/octet-stream]
Saving to: 'Python-3.6.1.tgz'

Python-3.6.1.tgz                  100%[=============================================================>]  21.50M  26.8MB/s    in 0.8s

2024-01-10 10:14:52 (26.8 MB/s) - 'Python-3.6.1.tgz' saved [22540566/22540566]

This output shows:

• The URL being accessed
• The IP address of the server
• The HTTP response (200 OK means successful)
• The file size (approximately 21MB)
• The download progress
• The download speed and time
• Confirmation that the file was saved with its original name

Let's verify that the file was downloaded correctly:

ls -lh

You should see the Python-3.6.1.tgz file in the directory with its corresponding size.

Downloading with Custom Filenames

In this step, we'll explore how to download a file and save it with a custom filename rather than using the default name from the URL.

Using the -O Option

The -O (capital letter O, not zero) option allows you to specify the name of the output file. This is useful when:

• You want a more descriptive filename
• The default filename from the URL is too complex
• You're downloading multiple versions of the same resource

Let's try downloading another Python package, but this time we'll save it with a custom name:

cd ~/project/download_resources
wget -O CustomPython.tgz https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz

In this command:

• -O CustomPython.tgz tells wget to save the file as CustomPython.tgz
• The file will be downloaded from the specified URL but saved with our custom name

The output will be similar to before, but notice the "Saving to" line now shows our custom filename:

--2024-01-10 10:20:51--  https://www.python.org/ftp/python/3.6.2/Python-3.6.2.tgz
Resolving www.python.org (www.python.org)... 151.101.76.223, 2a04:4e42:12::223
Connecting to www.python.org (www.python.org)|151.101.76.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22676538 (22M) [application/octet-stream]
Saving to: 'CustomPython.tgz'

CustomPython.tgz                  100%[=============================================================>]  21.63M  25.9MB/s    in 0.8s

2024-01-10 10:20:52 (25.9 MB/s) - 'CustomPython.tgz' saved [22676538/22676538]

Let's check that our file was downloaded with the custom name:

ls -lh

You should now see both the original Python-3.6.1.tgz file and your new CustomPython.tgz file in the directory.

Additional Useful Options

While we're learning about wget options, here are a few more useful ones:

• -q (quiet): Suppresses the output, useful for scripts
• -c (continue): Resumes a partially downloaded file
• --limit-rate=1m: Limits the download speed (e.g., to 1 megabyte per second)

For example, to download quietly with a custom name:

wget -q -O PythonQuiet.tgz https://www.python.org/ftp/python/3.6.3/Python-3.6.3.tgz

This command won't show any output, but the file will be downloaded. You can verify it with:

ls -lh

You should now see the PythonQuiet.tgz file in your directory as well.

Downloading Multiple Files from a List

In real-world scenarios, you often need to download multiple files. Typing each wget command manually would be inefficient. Fortunately, wget can download multiple files from a list, which is perfect for automation.

Creating a File with URLs

First, let's create a text file containing the URLs of the files we want to download:

cd ~/project/download_resources
echo "https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz" > download_list.txt
echo "https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz" >> download_list.txt
echo "https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz" >> download_list.txt

In these commands:

• The first echo command creates a new file called download_list.txt and adds the first URL
• The subsequent echo commands append additional URLs to the file using >> (double redirect)

Let's check the content of our file to make sure it's correct:

cat download_list.txt

You should see three URLs, each on its own line:

https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz

Using wget with an Input File

Now we can use the -i option with wget to read URLs from our file and download all the files:

wget -i download_list.txt

This command tells wget to read the URLs from download_list.txt and download each file in sequence. You'll see output for each download, similar to when you downloaded a single file:

--2024-01-10 10:30:51--  https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
Resolving www.python.org (www.python.org)... 151.101.76.223, 2a04:4e42:12::223
Connecting to www.python.org (www.python.org)|151.101.76.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22808518 (22M) [application/octet-stream]
Saving to: 'Python-3.7.0.tgz'

Python-3.7.0.tgz                  100%[=============================================================>]  21.75M  25.9MB/s    in 0.8s

2024-01-10 10:30:52 (25.9 MB/s) - 'Python-3.7.0.tgz' saved [22808518/22808518]

--2024-01-10 10:30:52--  https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
...

Verifying Downloaded Files

After the downloads complete, let's verify that all files were downloaded correctly:

ls -lh Python-3.7.*

You should see the three Python 3.7.x files we downloaded from the list:

-rw-r--r-- 1 labex labex 22M Jan 10 10:30 Python-3.7.0.tgz
-rw-r--r-- 1 labex labex 22M Jan 10 10:30 Python-3.7.1.tgz
-rw-r--r-- 1 labex labex 22M Jan 10 10:31 Python-3.7.2.tgz

Creating a Batch Download Script

For future use, let's create a simple shell script that can download files from a list. This demonstrates how wget can be used in automation:

cd ~/project/download_resources
nano batch_download.sh

Enter the following content in the file:

#!/bin/bash
## A simple script to download files from a list
if [ -f "$1" ]; then
  echo "Downloading files from list: $1"
  wget -i "$1"
else
  echo "Error: File $1 not found"
  exit 1
fi

Save the file by pressing Ctrl+O, then Enter, and exit with Ctrl+X.

Make the script executable:

chmod +x batch_download.sh

Now you can use this script to download files from any list in the future:

./batch_download.sh download_list.txt

This command would do the same thing as our earlier wget -i download_list.txt command, but it's wrapped in a script that you can reuse.

Summary

In this lab, you have learned how to use the wget command for non-interactive downloading in Linux. You now have the skills to:

• Download individual files using basic wget commands
• Save downloaded files with custom names using the -O option
• Create a list of URLs and download multiple files at once using the -i option
• Create simple automation scripts for batch downloading

These skills are valuable for system administrators, developers, and any Linux user who needs to download files efficiently from the command line. Non-interactive downloading is particularly useful for automation, remote server management, and situations where graphical interfaces are not available.

Some additional wget features that you might explore on your own include:

• Recursive downloading with -r for mirroring websites
• Background downloading with -b for long-running downloads
• Using authentication with --user and --password for protected resources
• Setting timeouts, retries, and other connection parameters

With these capabilities, you can handle a wide range of downloading scenarios efficiently from the Linux command line.