Downloading Multiple Files from a List
In real-world scenarios, you often need to download multiple files. Typing each wget
command manually would be inefficient. Fortunately, wget
can download multiple files from a list, which is perfect for automation.
Creating a File with URLs
First, let's create a text file containing the URLs of the files we want to download:
cd ~/project/download_resources
echo "https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz" > download_list.txt
echo "https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz" >> download_list.txt
echo "https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz" >> download_list.txt
In these commands:
- The first
echo
command creates a new file called download_list.txt
and adds the first URL
- The subsequent
echo
commands append additional URLs to the file using >>
(double redirect)
Let's check the content of our file to make sure it's correct:
cat download_list.txt
You should see three URLs, each on its own line:
https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
https://www.python.org/ftp/python/3.7.2/Python-3.7.2.tgz
Now we can use the -i
option with wget
to read URLs from our file and download all the files:
wget -i download_list.txt
This command tells wget
to read the URLs from download_list.txt
and download each file in sequence. You'll see output for each download, similar to when you downloaded a single file:
--2024-01-10 10:30:51-- https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tgz
Resolving www.python.org (www.python.org)... 151.101.76.223, 2a04:4e42:12::223
Connecting to www.python.org (www.python.org)|151.101.76.223|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22808518 (22M) [application/octet-stream]
Saving to: 'Python-3.7.0.tgz'
Python-3.7.0.tgz 100%[=============================================================>] 21.75M 25.9MB/s in 0.8s
2024-01-10 10:30:52 (25.9 MB/s) - 'Python-3.7.0.tgz' saved [22808518/22808518]
--2024-01-10 10:30:52-- https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tgz
...
Verifying Downloaded Files
After the downloads complete, let's verify that all files were downloaded correctly:
ls -lh Python-3.7.*
You should see the three Python 3.7.x files we downloaded from the list:
-rw-r--r-- 1 labex labex 22M Jan 10 10:30 Python-3.7.0.tgz
-rw-r--r-- 1 labex labex 22M Jan 10 10:30 Python-3.7.1.tgz
-rw-r--r-- 1 labex labex 22M Jan 10 10:31 Python-3.7.2.tgz
Creating a Batch Download Script
For future use, let's create a simple shell script that can download files from a list. This demonstrates how wget
can be used in automation:
cd ~/project/download_resources
nano batch_download.sh
Enter the following content in the file:
#!/bin/bash
## A simple script to download files from a list
if [ -f "$1" ]; then
echo "Downloading files from list: $1"
wget -i "$1"
else
echo "Error: File $1 not found"
exit 1
fi
Save the file by pressing Ctrl+O
, then Enter
, and exit with Ctrl+X
.
Make the script executable:
chmod +x batch_download.sh
Now you can use this script to download files from any list in the future:
./batch_download.sh download_list.txt
This command would do the same thing as our earlier wget -i download_list.txt
command, but it's wrapped in a script that you can reuse.