Linux expect Command with Practical Examples

LinuxLinuxBeginner
Practice Now

Introduction

In this lab, you will learn how to use the Linux expect command to automate interactive command-line applications. The expect command is a powerful automation tool that allows scripts to interact with programs that require user input, such as SSH, FTP, and other interactive programs.

By the end of this lab, you will be able to:

  • Understand the purpose and basic syntax of the expect command
  • Create scripts to automate SSH logins
  • Handle various prompts and responses in your expect scripts

The expect command can significantly reduce manual intervention for repetitive tasks, making system administration and routine tasks more efficient. You will start by installing and exploring the basic syntax of expect, then progress to creating scripts for automating SSH logins and handling various interactive prompts.

Linux Commands Cheat Sheet


Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL linux(("Linux")) -.-> linux/FileandDirectoryManagementGroup(["File and Directory Management"]) linux(("Linux")) -.-> linux/UserandGroupManagementGroup(["User and Group Management"]) linux(("Linux")) -.-> linux/PackagesandSoftwaresGroup(["Packages and Softwares"]) linux(("Linux")) -.-> linux/VersionControlandTextEditorsGroup(["Version Control and Text Editors"]) linux(("Linux")) -.-> linux/BasicSystemCommandsGroup(["Basic System Commands"]) linux(("Linux")) -.-> linux/BasicFileOperationsGroup(["Basic File Operations"]) linux(("Linux")) -.-> linux/SystemInformationandMonitoringGroup(["System Information and Monitoring"]) linux/BasicSystemCommandsGroup -.-> linux/echo("Text Display") linux/BasicFileOperationsGroup -.-> linux/ls("Content Listing") linux/BasicFileOperationsGroup -.-> linux/chmod("Permission Modifying") linux/FileandDirectoryManagementGroup -.-> linux/cd("Directory Changing") linux/FileandDirectoryManagementGroup -.-> linux/which("Command Locating") linux/SystemInformationandMonitoringGroup -.-> linux/date("Date/Time Displaying") linux/UserandGroupManagementGroup -.-> linux/useradd("User Adding") linux/PackagesandSoftwaresGroup -.-> linux/apt("Package Handling") linux/VersionControlandTextEditorsGroup -.-> linux/nano("Simple Text Editing") subgraph Lab Skills linux/echo -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/ls -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/chmod -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/cd -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/which -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/date -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/useradd -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/apt -.-> lab-422669{{"Linux expect Command with Practical Examples"}} linux/nano -.-> lab-422669{{"Linux expect Command with Practical Examples"}} end

Understanding the expect Command and Its Basic Syntax

The expect command in Linux allows you to automate interactive command-line programs that normally require user input. This is particularly useful for tasks like automated logins, file transfers, or any situation where a program prompts for input.

Installing expect

First, let us verify that the expect package is installed on our system. Open your terminal and run:

which expect

If expect is already installed, you will see its path (like /usr/bin/expect). If not, you will need to install it:

sudo apt-get update
sudo apt-get install -y expect

Understanding Basic expect Syntax

The expect command uses a scripting language based on Tcl (Tool Command Language). The basic structure of an expect script includes the following commands:

  1. spawn: Starts a process to interact with
  2. expect: Waits for specific output from the spawned process
  3. send: Sends input to the spawned process
  4. set timeout: Sets how long to wait for expected output

Let us create a simple expect script to demonstrate these concepts. Open a text editor and create a file named hello.exp in your project directory:

cd ~/project
nano hello.exp

Type the following content into the file:

#!/usr/bin/expect -f

## Set a timeout of 10 seconds
set timeout 10

## Spawn the bash process
spawn bash

## Wait for the bash prompt
expect "$ "

## Send a command to the bash process
send "echo Hello from expect\r"

## Wait for the bash prompt again
expect "$ "

## Exit the bash session
send "exit\r"

## Wait for the process to complete
expect eof

Save the file by pressing Ctrl+O, then Enter, and exit nano with Ctrl+X.

Make the script executable:

chmod +x ~/project/hello.exp

Now run the script:

~/project/hello.exp

You should see output similar to this:

spawn bash
$ echo Hello from expect
Hello from expect
$ exit
exit

Understanding Each Line of the Script

Let me explain what each line in the script does:

  • #!/usr/bin/expect -f: This is a shebang line that tells the system to use the expect interpreter to run this script.
  • set timeout 10: This sets a timeout of 10 seconds for any expect command that follows.
  • spawn bash: This starts a new bash shell process that expect will interact with.
  • expect "$ ": This waits for the bash prompt to appear.
  • send "echo Hello from expect\r": This sends the command to the bash shell. Note the \r at the end, which simulates pressing Enter.
  • expect "$ ": This waits for the bash prompt again, after the command has been executed.
  • send "exit\r": This sends the exit command to close the bash shell.
  • expect eof: This waits for the spawned process to terminate.

This simple example demonstrates the core functionality of expect. In the next steps, we will use these concepts to create more practical scripts.

Creating a Mock SSH Login Script with expect

In this step, we will create an expect script that simulates an SSH login process. Since we cannot perform an actual SSH login in this environment, we will create a mock script that demonstrates the principles.

Understanding SSH Authentication Flow

When connecting to a remote server via SSH, the typical interaction involves:

  1. Initiating the connection with ssh username@hostname
  2. Accepting the host key (if connecting for the first time)
  3. Entering your password when prompted
  4. Getting access to the remote shell

Let us create a mock SSH environment to demonstrate how expect can automate this process.

Creating a Mock SSH Server Script

First, let us create a script that simulates an SSH server prompting for a password:

cd ~/project
nano mock_ssh_server.sh

Enter the following content:

#!/bin/bash

echo "The authenticity of host 'mockserver' can't be established."
echo "RSA key fingerprint is SHA256:abcdefghijklmnopqrstuvwxyz123456."
echo "Are you sure you want to continue connecting (yes/no)? "
read answer

if [ "$answer" != "yes" ]; then
  echo "Host key verification failed."
  exit 1
fi

echo "Warning: Permanently added 'mockserver' (RSA) to the list of known hosts."
echo "Password: "
read -s password

if [ "$password" == "mockpassword" ]; then
  echo "Last login: Wed Nov 1 12:00:00 2023 from 192.168.1.100"
  echo "Welcome to Mock SSH Server"
  echo "mockuser@mockserver:~$ "

  while true; do
    read -p "" command
    if [ "$command" == "exit" ]; then
      echo "Connection to mockserver closed."
      exit 0
    else
      echo "Executing: $command"
      echo "mockuser@mockserver:~$ "
    fi
  done
else
  echo "Permission denied, please try again."
  exit 1
fi

Save the file and make it executable:

chmod +x ~/project/mock_ssh_server.sh

This script simulates:

  • The SSH host verification prompt
  • The password prompt
  • A simple shell that responds to commands

Creating an expect Script to Automate the Login

Now, let us create an expect script that automates the interaction with our mock SSH server:

cd ~/project
nano ssh_login.exp

Enter the following content:

#!/usr/bin/expect -f

## Set variables
set timeout 10
set password "mockpassword"

## Start the mock SSH server
spawn ./mock_ssh_server.sh

## Handle the host verification prompt
expect "Are you sure you want to continue connecting (yes/no)? "
send "yes\r"

## Handle the password prompt
expect "Password: "
send "$password\r"

## Wait for the shell prompt
expect "mockuser@mockserver:~$ "

## Execute a command
send "ls -la\r"
expect "mockuser@mockserver:~$ "

## Exit the session
send "exit\r"

## Wait for the process to complete
expect eof

puts "\nSSH login automation completed successfully!"

Save the file and make it executable:

chmod +x ~/project/ssh_login.exp

Running the Automated Login Script

Now, let us run our expect script to automate the interaction with the mock SSH server:

cd ~/project
./ssh_login.exp

You should see output similar to this:

spawn ./mock_ssh_server.sh
The authenticity of host 'mockserver' can't be established.
RSA key fingerprint is SHA256:abcdefghijklmnopqrstuvwxyz123456.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mockserver' (RSA) to the list of known hosts.
Password:
Last login: Wed Nov 1 12:00:00 2023 from 192.168.1.100
Welcome to Mock SSH Server
mockuser@mockserver:~$ ls -la
Executing: ls -la
mockuser@mockserver:~$ exit
Connection to mockserver closed.

SSH login automation completed successfully!

Explaining the Script

Let me explain what each part of our expect script does:

  1. set timeout 10: Sets a global timeout of 10 seconds for all expect commands.
  2. set password "mockpassword": Stores the password in a variable.
  3. spawn ./mock_ssh_server.sh: Starts our mock SSH server script.
  4. expect "Are you sure you want to continue connecting (yes/no)? ": Waits for the host verification prompt.
  5. send "yes\r": Sends "yes" to accept the host key.
  6. expect "Password: ": Waits for the password prompt.
  7. send "$password\r": Sends the password.
  8. expect "mockuser@mockserver:~$ ": Waits for the shell prompt.
  9. send "ls -la\r": Sends a command to list files.
  10. expect "mockuser@mockserver:~$ ": Waits for the shell prompt again.
  11. send "exit\r": Sends the exit command to close the session.
  12. expect eof: Waits for the process to terminate.
  13. puts "\nSSH login automation completed successfully!": Prints a success message.

This example demonstrates how expect can be used to automate the entire SSH login process, from accepting the host key to executing commands on the remote server and safely exiting.

Handling Multiple Prompts and Responses with expect

In real-world scenarios, interactive programs often present multiple prompts and may require different responses based on different conditions. In this step, we will learn how to handle multiple prompts and conditional responses in expect scripts.

Understanding Conditional Expect Blocks

The expect command can be used with a pattern-action block structure to handle different possible prompts:

expect {
    "pattern1" { actions for pattern1 }
    "pattern2" { actions for pattern2 }
    timeout { actions for timeout }
    eof { actions for end of file }
}

This structure allows your script to respond differently depending on what output it receives.

Creating a Multi-Prompt Handling Script

Let us create a script that simulates a program with multiple prompts:

cd ~/project
nano multi_prompt.sh

Enter the following content:

#!/bin/bash

echo "Please select an option:"
echo "1. Show date and time"
echo "2. List files"
echo "3. Show system info"
echo "4. Exit"
echo -n "Enter your choice (1-4): "
read choice

case $choice in
  1)
    echo "Current date and time:"
    date
    ;;
  2)
    echo "File listing:"
    ls -la
    ;;
  3)
    echo "System information:"
    uname -a
    ;;
  4)
    echo "Exiting program..."
    exit 0
    ;;
  *)
    echo "Invalid option. Please enter a number between 1 and 4."
    exit 1
    ;;
esac

echo "Do you want to continue? (yes/no): "
read answer

if [ "$answer" == "yes" ]; then
  echo "Continuing..."
  echo "Operation completed successfully."
else
  echo "Exiting program..."
fi

Save the file and make it executable:

chmod +x ~/project/multi_prompt.sh

Now, let us create an expect script to interact with this program and handle all possible prompts:

cd ~/project
nano handle_prompts.exp

Enter the following content:

#!/usr/bin/expect -f

## Set a timeout
set timeout 10

## Start the multi-prompt program
spawn ./multi_prompt.sh

## Wait for the choice prompt
expect "Enter your choice (1-4): "

## Generate a random choice (1-3)
set choice [expr {int(rand() * 3) + 1}]
send "$choice\r"

## Process the result based on the choice
switch $choice {
    1 {
        expect "Current date and time:"
        expect "Do you want to continue? (yes/no): "
    }
    2 {
        expect "File listing:"
        expect "Do you want to continue? (yes/no): "
    }
    3 {
        expect "System information:"
        expect "Do you want to continue? (yes/no): "
    }
}

## Handle the continue prompt
expect {
    "Do you want to continue? (yes/no): " {
        ## 70% chance to say yes, 30% chance to say no
        if {rand() < 0.7} {
            send "yes\r"
            expect "Operation completed successfully."
        } else {
            send "no\r"
            expect "Exiting program..."
        }
    }
    timeout {
        puts "Timeout waiting for continue prompt"
        exit 1
    }
}

## Wait for the program to complete
expect eof

puts "\nMulti-prompt handling completed successfully!"

Save the file and make it executable:

chmod +x ~/project/handle_prompts.exp

Running the Multi-Prompt Handling Script

Now, let us run our expect script to interact with the multi-prompt program:

cd ~/project
./handle_prompts.exp

Each time you run this script, it will randomly select one of the options and randomly decide whether to continue or exit. Here is an example output:

spawn ./multi_prompt.sh
Please select an option:
1. Show date and time
2. List files
3. Show system info
4. Exit
Enter your choice (1-4): 2
File listing:
total 20
drwxr-xr-x 2 labex labex 4096 Nov  1 10:00 .
drwxr-xr-x 4 labex labex 4096 Nov  1 10:00 ..
-rwxr-xr-x 1 labex labex  345 Nov  1 10:00 handle_prompts.exp
-rwxr-xr-x 1 labex labex  578 Nov  1 10:00 multi_prompt.sh
-rwxr-xr-x 1 labex labex  221 Nov  1 10:00 ssh_login.exp
Do you want to continue? (yes/no): yes
Continuing...
Operation completed successfully.

Multi-prompt handling completed successfully!

Creating a More Advanced expect Script

Now let us create a more advanced script that can handle unexpected prompts and errors:

cd ~/project
nano advanced_expect.exp

Enter the following content:

#!/usr/bin/expect -f

## Set a timeout
set timeout 10

## Define variables
set program "./multi_prompt.sh"
set max_retries 3
set retry_count 0

## Define a procedure to handle errors
proc handle_error {} {
    global retry_count max_retries program
    incr retry_count

    if {$retry_count < $max_retries} {
        puts "\nRetrying... Attempt $retry_count of $max_retries"
        ## Start the program again
        spawn $program
        return 1
    } else {
        puts "\nMaximum retry attempts reached. Exiting."
        exit 1
    }
}

## Start the program
spawn $program

## Main interaction loop
while {$retry_count < $max_retries} {
    expect {
        "Enter your choice (1-4): " {
            send "1\r"  ## Always choose option 1 for deterministic behavior
        }
        "Invalid option" {
            puts "\nReceived invalid option message."
            if {[handle_error]} continue
        }
        "Current date and time:" {
            ## Successfully got date output
        }
        "Do you want to continue? (yes/no): " {
            send "yes\r"
        }
        "Operation completed successfully." {
            puts "\nAdvanced expect script completed successfully!"
            break
        }
        timeout {
            puts "\nTimeout occurred waiting for prompt."
            if {[handle_error]} continue
        }
        eof {
            puts "\nUnexpected end of file."
            if {[handle_error]} continue
        }
    }
}

## Wait for the program to complete
expect eof

Save the file and make it executable:

chmod +x ~/project/advanced_expect.exp

Run the advanced script:

cd ~/project
./advanced_expect.exp

Example output:

spawn ./multi_prompt.sh
Please select an option:
1. Show date and time
2. List files
3. Show system info
4. Exit
Enter your choice (1-4): 1
Current date and time:
Wed Nov  1 10:00:00 UTC 2023
Do you want to continue? (yes/no): yes
Continuing...
Operation completed successfully.

Advanced expect script completed successfully!

Understanding the Advanced Script

This advanced script demonstrates several important expect techniques:

  1. Error handling: It uses a retry mechanism to handle errors or unexpected responses.
  2. Procedures: It defines a custom procedure called handle_error for reusable error handling.
  3. Control flow: It uses a while loop to maintain the interaction until success or maximum retries.
  4. Multiple expect patterns: It handles multiple different patterns and takes appropriate actions for each.
  5. Pattern order: The order of patterns in the expect block matters - more specific patterns should come before more general ones.

These techniques can be applied to automate complex interactive programs where the flow might vary or errors might occur.

Creating Practical expect Scripts for Common Tasks

In this step, we will create practical expect scripts for common tasks that system administrators often need to automate. We will focus on file operations, user interactions, and system monitoring.

Automating File Transfer with expect

Let us create an expect script that automates the transfer of a file using the scp command. Since we cannot perform an actual file transfer in this environment, we will simulate it:

cd ~/project
nano file_transfer.sh

Enter the following content to simulate an SCP-like file transfer:

#!/bin/bash

echo "scp file transfer simulation"
echo "Source file: $1"
echo "Destination: $2"
echo "Password: "
read -s password

if [ "$password" == "transfer123" ]; then
  echo "Transferring file..."
  echo "0%"
  sleep 1
  echo "25%"
  sleep 1
  echo "50%"
  sleep 1
  echo "75%"
  sleep 1
  echo "100%"
  echo "File transfer completed successfully."
else
  echo "Authentication failed."
  exit 1
fi

Save the file and make it executable:

chmod +x ~/project/file_transfer.sh

Now, let us create an expect script to automate this file transfer:

cd ~/project
nano file_transfer.exp

Enter the following content:

#!/usr/bin/expect -f

## Set variables
set timeout 10
set source_file "local_file.txt"
set destination "user@remote:/path/to/destination/"
set password "transfer123"

## Create a dummy source file
spawn bash -c "echo 'This is a test file' > $source_file"
expect eof

## Start the file transfer simulation
spawn ./file_transfer.sh $source_file $destination

## Handle the password prompt
expect "Password: "
send "$password\r"

## Monitor the transfer progress
expect "0%"
puts "Transfer started..."

expect "25%"
puts "Transfer 1/4 complete..."

expect "50%"
puts "Transfer 1/2 complete..."

expect "75%"
puts "Transfer 3/4 complete..."

expect "100%"
puts "Transfer almost done..."

expect "File transfer completed successfully."
puts "File transfer automation completed!"

## Clean up the dummy file
spawn bash -c "rm $source_file"
expect eof

Save the file and make it executable:

chmod +x ~/project/file_transfer.exp

Run the file transfer automation script:

cd ~/project
./file_transfer.exp

Example output:

spawn bash -c echo 'This is a test file' > local_file.txt
spawn ./file_transfer.sh local_file.txt user@remote:/path/to/destination/
scp file transfer simulation
Source file: local_file.txt
Destination: user@remote:/path/to/destination/
Password:
Transferring file...
0%
Transfer started...
25%
Transfer 1/4 complete...
50%
Transfer 1/2 complete...
75%
Transfer 3/4 complete...
100%
Transfer almost done...
File transfer completed successfully.
File transfer automation completed!
spawn bash -c rm local_file.txt

Automating User Creation with expect

Now, let us create an expect script that automates user creation. Again, we will simulate this process:

cd ~/project
nano create_user.sh

Enter the following content:

#!/bin/bash

echo "User creation utility"
echo "Please enter new username: "
read username

echo "Please enter password for $username: "
read -s password

echo "Please confirm password: "
read -s password_confirm

if [ "$password" != "$password_confirm" ]; then
  echo "Error: Passwords do not match."
  exit 1
fi

echo "Creating user $username..."
echo "User $username created successfully."
echo "Do you want to add this user to the sudo group? (yes/no): "
read sudo_choice

if [ "$sudo_choice" == "yes" ]; then
  echo "Adding $username to sudo group..."
  echo "User $username added to sudo group."
fi

echo "User setup completed."

Save the file and make it executable:

chmod +x ~/project/create_user.sh

Now, let us create an expect script to automate user creation:

cd ~/project
nano create_user.exp

Enter the following content:

#!/usr/bin/expect -f

## Set variables
set timeout 10
set username "testuser"
set password "P@ssw0rd123"
set add_sudo "yes"

## Start the user creation utility
spawn ./create_user.sh

## Handle the username prompt
expect "Please enter new username: "
send "$username\r"

## Handle the password prompt
expect "Please enter password for $username: "
send "$password\r"

## Handle the password confirmation prompt
expect "Please confirm password: "
send "$password\r"

## Wait for the user creation confirmation
expect "User $username created successfully."

## Handle the sudo prompt
expect "Do you want to add this user to the sudo group? (yes/no): "
send "$add_sudo\r"

## If we chose to add to sudo, wait for confirmation
if {$add_sudo == "yes"} {
    expect "User $username added to sudo group."
}

## Wait for completion
expect "User setup completed."

puts "\nUser creation automation completed successfully!"

Save the file and make it executable:

chmod +x ~/project/create_user.exp

Run the user creation automation script:

cd ~/project
./create_user.exp

Example output:

spawn ./create_user.sh
User creation utility
Please enter new username:
testuser
Please enter password for testuser:
Please confirm password:
Creating user testuser...
User testuser created successfully.
Do you want to add this user to the sudo group? (yes/no):
yes
Adding testuser to sudo group...
User testuser added to sudo group.
User setup completed.

User creation automation completed successfully!

Understanding Practical expect Scripts

The practical scripts we created demonstrate several important concepts for real-world automation:

  1. Sequential Interaction: Both scripts follow a defined sequence of prompts and responses.
  2. Progress Monitoring: The file transfer script monitors progress and provides user-friendly updates.
  3. Conditional Logic: The user creation script uses conditional logic to handle the sudo option.
  4. Environment Setup and Cleanup: The file transfer script creates and cleans up test files.

These techniques can be applied to automate many common system administration tasks, such as:

  • Remote backups
  • Software installations
  • System configuration
  • Batch operations

By mastering expect, you can automate complex interactive processes that would otherwise require manual intervention, saving time and reducing the potential for human error.

Summary

In this lab, you have learned how to use the expect command in Linux to automate interactive command-line applications. You have gained practical experience with:

  • Installing and understanding the basic syntax of the expect command
  • Creating scripts to automate SSH logins and handle various authentication prompts
  • Handling multiple prompts and responses in expect scripts
  • Creating practical automation scripts for common system administration tasks

The expect command is a powerful tool for system administrators and developers who need to automate interactive processes. By using expect, you can eliminate the need for manual intervention in repetitive tasks, saving time and reducing the risk of human error.

Some key takeaways from this lab:

  • The expect command uses a pattern-action model to interact with programs
  • Scripts can be made more robust by handling various possible prompts and error conditions
  • Complex interactions can be automated using conditional logic and custom procedures
  • Practical automation can significantly improve efficiency for common system tasks

With the skills you have learned in this lab, you can now create your own automation scripts for various interactive command-line applications.

Linux Commands Cheat Sheet