LabEx
Log InJoin Free

Linux URL Data Transferring

LinuxBeginner
Practice Now

Introduction

Welcome to the Linux URL Data Transferring lab. In this lab, you will learn how to transfer data using URLs in Linux systems. This is a fundamental skill for system administrators, developers, and security professionals.

The primary tool we will learn is curl, a powerful command-line utility for transferring data with URL syntax. You will learn how to retrieve data from websites, save content to files, use different HTTP methods, and customize your requests with various options.

By the end of this lab, you will have practical experience with essential data transfer techniques that are commonly used in real-world scenarios.

Understanding curl Basics

curl is a command-line tool for transferring data with URL syntax. It supports numerous protocols including HTTP, HTTPS, FTP, and many more. In this step, you will learn the basic usage of curl to retrieve web content.

Let's begin by opening a terminal. The terminal should already be at the directory /home/labex/project. If not, you can change to this directory using:

cd ~/project

Your First curl Command

Let's start by using curl to retrieve content from a website. Type the following command:

curl http://example.com

After running this command, you should see the HTML content of example.com displayed in your terminal. This output includes HTML tags and all the content of the webpage. The output should look similar to this:

<!doctype html>
<html>
<head>
    <title>Example Domain</title>
    ...
</head>
<body>
    <div>
        <h1>Example Domain</h1>
        <p>This domain is for use in illustrative examples in documents...</p>
        ...
    </div>
</body>
</html>

This demonstrates the most basic usage of curl - retrieving the content of a URL and displaying it directly in the terminal.

Understanding the Output

The output you see is the raw HTML response from the server. When you visit a website using a browser, the browser interprets this HTML and renders it as a formatted webpage. However, curl simply shows you the raw content.

Let's try another example to see different content. Run:

curl https://httpbin.org/json

This time, you should see a JSON response. The output should look like:

{
  "slideshow": {
    "author": "Yours Truly",
    "date": "date of publication",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!",
        "type": "all"
      },
      {
        "items": [
          "Why <em>WonderWidgets</em> are great",
          "Who <em>buys</em> WonderWidgets"
        ],
        "title": "Overview",
        "type": "all"
      }
    ],
    "title": "Sample Slide Show"
  }
}

As you can see, curl can retrieve different types of content depending on what the server provides.

Saving Output to Files

In many cases, you might want to save the content you retrieve rather than just viewing it in the terminal. In this step, you will learn how to save curl output to files.

Saving Using Output Redirection

The simplest way to save the output of a curl command is by using redirection with the > operator. Run the following command:

curl http://example.com > example.html

This command retrieves the content from example.com and saves it to a file named example.html in your current directory. To verify that the file was created, run:

ls -l example.html

You should see the file listed with its size and other details.

To check the content of the file, you can use the cat command:

cat example.html

You should see the same HTML content that was previously displayed in the terminal.

Using curl's -o Option

curl provides a specific option -o (or --output) to save the retrieved content to a file. This is particularly useful when the output redirection might cause issues. Let's try it:

curl -o httpbin_data.json https://httpbin.org/json

This command saves the JSON data from httpbin.org to a file named httpbin_data.json. Verify the file's creation with:

ls -l httpbin_data.json

And check its content with:

cat httpbin_data.json

Creating a Data Directory

It's good practice to organize your files. Let's create a dedicated data directory (if it doesn't already exist) and save some content there:

mkdir -p data
curl -o data/google_data.txt http://www.google.com

The mkdir -p command creates the data directory if it doesn't exist yet. The -p flag ensures that no error is reported if the directory already exists.

Now, verify that the file was created in the data directory:

ls -l data/google_data.txt

You should see the file listed with its details.

Advanced curl Options

curl offers many options that allow you to customize your requests. In this step, you will learn about some of the most useful options.

Verbose Mode

The -v (or --verbose) option tells curl to display detailed information about the request and response. This is particularly useful for debugging. Run:

curl -v http://example.com

You should see a lot of information, including:

  • The request headers sent by curl
  • Connection details
  • Response headers from the server
  • The actual content of the response

The output will look similar to this (though some details will vary):

*   Trying 93.184.216.34:80...
* Connected to example.com (93.184.216.34) port 80 (#0)
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Age: 578793
< Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 26 Dec 2023 13:45:32 GMT
< Etag: "3147526947+ident"
< Expires: Tue, 02 Jan 2024 13:45:32 GMT
< Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT
< Server: ECS (dcb/7F83)
< Vary: Accept-Encoding
< X-Cache: HIT
< Content-Length: 1256
<
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
    ...

Lines beginning with * are informational messages from curl itself. Lines beginning with > are request headers. Lines beginning with < are response headers. The actual content follows the response headers.

Custom Headers

Sometimes you need to send specific headers with your request. You can use the -H (or --header) option for this. Let's try setting a custom User-Agent:

curl -H "User-Agent: MyCustomClient/1.0" https://httpbin.org/user-agent

This command sends a request to httpbin.org with a custom User-Agent header. The response should confirm that your custom User-Agent was received:

{
  "user-agent": "MyCustomClient/1.0"
}

Following Redirects

By default, curl doesn't follow HTTP redirects. You can use the -L (or --location) option to make it follow redirects. Let's try:

curl -L http://github.com

Without the -L option, you would just get a redirect response. With -L, curl follows the redirect and shows you the content of the final destination.

Let's create a file in the data directory to practice with these options:

curl -v -L -o data/redirect_example.txt http://github.com

This command:

  1. Uses verbose mode (-v)
  2. Follows redirects (-L)
  3. Saves the output to data/redirect_example.txt
  4. Requests the content from github.com

Verify that the file was created:

ls -l data/redirect_example.txt

You should see the file listed with its details.

HTTP Methods and Data Sending

curl supports various HTTP methods beyond the default GET method. In this step, you will learn how to use different HTTP methods and send data with your requests.

Using the POST Method

The POST method is commonly used to submit data to a server. You can use the -X option to specify the HTTP method and the -d option to provide the data. Let's try:

curl -X POST -d "name=John&age=25" https://httpbin.org/post

This command sends a POST request with form data to httpbin.org. The response should echo back the data you sent:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "age": "25",
    "name": "John"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "16",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-65b..."
  },
  "json": null,
  "origin": "your-ip-address",
  "url": "https://httpbin.org/post"
}

Sending JSON Data

To send JSON data, you need to specify the Content-Type header. Run:

curl -X POST -H "Content-Type: application/json" -d '{"name":"John","age":25}' https://httpbin.org/post

The response should include your JSON data:

{
  "args": {},
  "data": "{\"name\":\"John\",\"age\":25}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "24",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-65b..."
  },
  "json": {
    "age": 25,
    "name": "John"
  },
  "origin": "your-ip-address",
  "url": "https://httpbin.org/post"
}

Using Other HTTP Methods

curl supports all standard HTTP methods. Let's try a PUT request:

curl -X PUT -d "data=example" https://httpbin.org/put

You can also try a DELETE request:

curl -X DELETE https://httpbin.org/delete

Each of these commands will return a response showing the details of your request.

Let's save the output of a POST request to a file:

curl -X POST -d "name=John&age=25" -o data/post_response.json https://httpbin.org/post

Verify that the file was created:

ls -l data/post_response.json

And check its content:

cat data/post_response.json

You should see the JSON response from httpbin.org, similar to what was shown earlier.

Downloading Files and Monitoring Progress

curl is excellent for downloading files from the internet. In this step, you will learn how to download files and monitor the download progress.

Basic File Download

To download a file, you can use the -o option we learned earlier. Let's download a sample image:

curl -o data/sample.jpg https://httpbin.org/image/jpeg

This command downloads a JPEG image from httpbin.org and saves it as sample.jpg in the data directory.

Verify that the file was downloaded:

ls -l data/sample.jpg

Downloading with Progress Bar

For larger downloads, it's useful to see the progress. You can use the -# option to display a progress bar:

curl -## -o data/sample_with_progress.jpg https://httpbin.org/image/jpeg

You should see a progress bar like this:

######################################################################### 100.0%

Using the Output File Option with Automatic Name

If you want to use the same filename as the remote file, you can use the -O (capital O) option:

cd data
curl -O https://httpbin.org/image/jpeg
cd ..

This downloads the file and saves it as jpeg (the last part of the URL) in the data directory.

Verify that the file was downloaded:

ls -l data/jpeg

Resuming Interrupted Downloads

If a download is interrupted, you can resume it using the -C - option:

curl -C - -o data/resume_example.jpg https://httpbin.org/image/jpeg

This tells curl to automatically determine where to resume the download from.

Limiting Download Speed

You can limit the download speed using the --limit-rate option:

curl --limit-rate 100K -o data/rate_limited.jpg https://httpbin.org/image/jpeg

This limits the download speed to 100 KB/s.

Downloading Multiple Files

You can download multiple files with a single command:

curl -o data/image1.jpg https://httpbin.org/image/jpeg -o data/image2.png https://httpbin.org/image/png

This downloads a JPEG and a PNG image and saves them as image1.jpg and image2.png respectively.

Verify that both files were downloaded:

ls -l data/image1.jpg data/image2.png

Summary

In this lab, you have learned how to use curl, a powerful command-line tool for transferring data with URLs. You now have practical experience with:

  1. Basic usage of curl for retrieving web content
  2. Saving curl output to files using redirection and the -o option
  3. Using advanced curl options like verbose mode (-v), custom headers (-H), and following redirects (-L)
  4. Working with different HTTP methods (GET, POST, PUT, DELETE) and sending data
  5. Downloading files and monitoring download progress

These skills are essential for many tasks in system administration, web development, API testing, and automation scripts. The curl command is widely used in various technical fields and is available on virtually all Linux distributions, macOS, and even Windows.

As you continue your journey in Linux and networking, remember that curl has many more options and capabilities than we covered in this lab. You can always refer to the curl manual (man curl) for more information.

Related Linux Courses
Quick Start with Linux

Quick Start with Linux

linux
Become a Junior System Administrator

Become a Junior System Administrator

linuxshell
Linux for Noobs

Linux for Noobs

linux