Executing Shell Commands

Introduction

In operations work, you often still need to call existing system commands even when writing Python. That is why understanding Python's process-execution tools is important.

The subprocess module lets Python launch other programs, capture their output, and inspect success or failure safely. It is the standard way to bridge Python code with shell commands.

In this lab, you will learn how to:

Import the subprocess module.
Run a simple Linux command (ls -l) using Python.
Capture the output of the command for further processing.
Decode the output from bytes to a readable string.
Handle errors gracefully when a command fails.

You will be working in the ~/project directory on a Linux environment.

Import Subprocess

In this step, you will create a Python script and import the necessary module to interact with the system shell.

The subprocess module is part of Python's standard library, so you don't need to install anything extra. It allows your Python script to talk to the underlying operating system.

This is the standard bridge between Python code and existing Unix tools. In operations work, that bridge matters because you rarely replace every shell command at once. More often, Python becomes the coordinator that calls system tools, captures their results, and adds higher-level logic around them.

First, ensure you are in the correct directory:

cd ~/project

Now, create a new Python file named list_files.py using the nano editor or your preferred method:

nano list_files.py

Add the following line to the top of the file to import the module:

import subprocess

print("Module imported successfully")

Save the file and exit the editor (in nano, press Ctrl+X, then Y, then Enter).

Now, run the script to make sure everything is working correctly:

python3 list_files.py

You should see the message "Module imported successfully".

Run a Command

In this step, you will modify your script to actually execute a Linux command. We will use the subprocess.run() function, which was introduced in Python 3.5 as the recommended approach for invoking subprocesses.

When Python launches another program, it creates a separate process. That child process has its own exit code, standard output, and standard error, just like a command you run directly in the shell. Understanding that process boundary makes later error handling much easier to reason about.

Open list_files.py again:

nano list_files.py

Modify the file to look like this. We will use the ls -l command to list files in the current directory with details:

import subprocess

## Run the 'ls -l' command
## The command and its arguments are passed as a list of strings
subprocess.run(["ls", "-l"])

Notice that we pass the command as a list: ["ls", "-l"]. This is safer and cleaner than passing a single string because it avoids issues with shell injection and spaces in filenames.

Save and exit the file.

Now, run the script:

python3 list_files.py

You should see the output of the ls -l command printed directly to your terminal. This happens because, by default, subprocess.run sends the command's output to the standard output (your screen).

Capture Output

In the previous step, the output was printed directly to the screen. However, in automation scripts, you usually want to capture that output into a variable so your Python code can process it (e.g., parse filenames, check for errors).

This is the difference between merely running a command and integrating it into a program. Once output is captured as data, your script can make decisions based on it instead of leaving the result only on the terminal for a human to read.

To do this, we need to use the capture_output=True argument.

Open list_files.py again:

nano list_files.py

Update the code to capture the result:

import subprocess

## Run the command and capture the output
result = subprocess.run(["ls", "-l"], capture_output=True)

## Print the captured result object
print(result)

Save and exit.

Run the script:

python3 list_files.py

You will see output that looks something like this:

CompletedProcess(args=['ls', '-l'], returncode=0, stdout=b'total 0\n-rw-rw-r-- 1 labex labex 0 ...', stderr=b'')

This is a CompletedProcess object. Notice the stdout=b'...' part. The b stands for bytes. This means Python has captured the raw data exactly as the computer sees it, not as readable text (string). We will convert this into human-readable text in the next step.

Decode Output

Since the output is captured as bytes, we need to decode it into a standard Python string (UTF-8) to make it readable and usable.

The bytes-versus-text distinction is fundamental in Python. Computers ultimately move raw bytes around, while humans usually want text. Many confusing bugs in automation come from forgetting which form you currently have.

Open list_files.py one more time:

nano list_files.py

Update the code to access the stdout attribute and decode it:

import subprocess

## Run the command and capture the output
result = subprocess.run(["ls", "-l"], capture_output=True)

## Decode the bytes to a string
output_string = result.stdout.decode("utf-8")

## Print the clean string
print("--- Command Output ---")
print(output_string)

Save and exit.

Run the script:

python3 list_files.py

Now you should see the clean, formatted output of the ls -l command, just as if you ran it in the shell directly, but now it's being printed by Python!

Note: Alternatively, you can pass text=True (or universal_newlines=True) to subprocess.run() to have Python handle the decoding automatically, but understanding the .decode() method is important for handling binary data.

Handle Errors

What happens if the command fails? For example, if you try to list a directory that doesn't exist. Good scripts need to handle these errors.

In Unix-like systems, command failure is usually communicated through a non-zero exit status. subprocess.run(..., check=True) translates that operating-system convention into a Python exception, which lets your script react using normal Python control flow.

Let's create a new script named handle_error.py to demonstrate this.

nano handle_error.py

Add the following code. We will try to list a non-existent directory named /nonexistent. We also use check=True, which tells Python to raise a CalledProcessError exception if the command returns a non-zero exit code (which indicates failure).

import subprocess

try:
    ## Try to list a directory that does not exist
    ## check=True raises an exception if the command fails
    subprocess.run(["ls", "/nonexistent"], check=True, capture_output=True)

except subprocess.CalledProcessError as e:
    print("An error occurred!")
    print(f"Return code: {e.returncode}")
    ## We can also print the error message generated by the command (stderr)
    print(f"Error message: {e.stderr.decode('utf-8')}")

Save and exit.

Run the script:

python3 handle_error.py

You should see output indicating that the error was caught:

An error occurred!
Return code: 2
Error message: ls: cannot access '/nonexistent': No such file or directory

If we didn't use the try-except block, the program would crash with a messy traceback. This approach allows your script to detect failures and decide what to do next (e.g., log the error, retry, or exit cleanly).

Summary

In this lab, you have learned the fundamentals of executing shell commands using Python's subprocess module. You have practiced:

Importing the subprocess module.
Running commands using subprocess.run().
Capturing standard output using capture_output=True.
Decoding byte output into readable strings.
Handling command failures using check=True and try-except blocks.

These skills are essential for building automation scripts, deployment tools, and system management utilities in Python.