How to fix 'UNREACHABLE!' error in Ansible

Introduction

Ansible is a powerful infrastructure automation tool that simplifies the management of complex IT environments. However, users often encounter the 'UNREACHABLE!' error, which can disrupt automation workflows. This error typically occurs when Ansible cannot establish a connection with the target hosts. In this lab, you will learn how to identify, troubleshoot, and prevent the 'UNREACHABLE!' error in your Ansible deployments.

By the end of this lab, you will understand common causes of connectivity issues in Ansible and be able to implement effective solutions to ensure your automation runs smoothly.

Skills Graph

Setting Up the Ansible Environment

In this step, we will set up a basic Ansible environment to work with. We'll install Ansible, configure the essential files, and ensure everything is ready for our experimentation.

Installing Ansible

First, let's install Ansible on the LabEx VM using the following commands:

sudo apt update
sudo apt install -y ansible

This will install the latest version of Ansible available in the Ubuntu repositories. Once the installation is complete, verify the installation by checking the Ansible version:

ansible --version

You should see output similar to the following, showing the Ansible version and configuration details:

ansible [core 2.12.x]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/labex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/labex/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.x (default, Apr 8 2022, 09:04:19) [GCC 11.2.0]
  jinja version = 3.0.3
  libyaml = True

Creating a Working Directory

Let's create a dedicated directory for our Ansible work:

mkdir -p ~/project/ansible-lab
cd ~/project/ansible-lab

Creating the Ansible Configuration File

Now, let's create a basic Ansible configuration file in our project directory:

cat > ansible.cfg << 'EOF'
[defaults]
inventory = ./inventory
host_key_checking = False
remote_user = labex
EOF

This configuration file:

• Specifies the location of our inventory file
• Disables SSH host key checking (useful for lab environments)
• Sets the default remote user to 'labex'

Creating the Inventory File

The inventory file defines the hosts that Ansible will manage. Let's create a simple inventory file:

cat > inventory << 'EOF'
[local]
localhost ansible_connection=local

[virtual]
virtual-host ansible_host=10.10.10.10
EOF

This inventory contains two groups:

• local: Contains only localhost, which uses a local connection
• virtual: Contains a virtual host that we'll use to demonstrate the 'UNREACHABLE!' error

The virtual-host is configured with an IP address (10.10.10.10) that doesn't exist in our environment, which will help us generate the 'UNREACHABLE!' error.

Testing Ansible

Let's test our Ansible setup by running a simple ping command against the local host:

ansible local -m ping

You should see a successful response like:

localhost | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

This confirms that Ansible is working correctly for the local connection. Now, let's try to ping the virtual host, which should fail:

ansible virtual -m ping

This will produce the 'UNREACHABLE!' error because the host doesn't exist:

virtual-host | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.10.10.10 port 22: Connection timed out",
    "unreachable": true
}

You have now successfully set up Ansible and created a scenario where the 'UNREACHABLE!' error occurs, which we'll investigate in the next step.

Understanding the 'UNREACHABLE!' Error

In the previous step, we encountered the 'UNREACHABLE!' error when trying to connect to a non-existent host. Now, let's understand the error in more detail and explore the common causes.

Analyzing the Error Message

Let's look at the error message we received:

virtual-host | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.10.10.10 port 22: Connection timed out",
    "unreachable": true
}

The error message provides valuable information:

• UNREACHABLE! indicates that Ansible couldn't establish a connection to the host
• The msg field tells us why: "Failed to connect to the host via ssh"
• The specific error is "Connection timed out," which means Ansible tried to connect but received no response

Common Causes of 'UNREACHABLE!' Errors

The 'UNREACHABLE!' error can occur for several reasons:

1. Network Issues: The host might be behind a firewall, or there could be network connectivity problems.
2. Incorrect Host Information: The hostname or IP address in the inventory might be wrong.
3. SSH Configuration: SSH might not be configured correctly on the target host.
4. Authentication Problems: The SSH key or password might be incorrect.
5. Host Unavailability: The host might be down or unreachable.

Creating a Test Playbook

Let's create a simple playbook to further demonstrate the error:

cat > test_playbook.yml << 'EOF'
---
- name: Test Connectivity
  hosts: all
  gather_facts: no
  tasks:
    - name: Ping the hosts
      ping:
EOF

This playbook attempts to ping all hosts defined in our inventory. Let's run it:

ansible-playbook test_playbook.yml

You should see output similar to:

PLAY [Test Connectivity] ************************************************

TASK [Ping the hosts] ***************************************************
ok: [localhost]
fatal: [virtual-host]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.10.10.10 port 22: Connection timed out", "unreachable": true}

PLAY RECAP *************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
virtual-host               : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

The playbook succeeded for localhost but failed for virtual-host with the 'UNREACHABLE!' error.

Inspecting Ansible Verbosity Levels

Ansible provides different verbosity levels to help diagnose issues. Let's try running the playbook with increased verbosity:

ansible-playbook test_playbook.yml -v

For even more detailed output, use -vv or -vvv:

ansible-playbook test_playbook.yml -vvv

The -vvv option provides the most detailed output, showing the exact SSH commands Ansible is trying to use:

<virtual-host> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o User=labex -o ConnectTimeout=10 -o ControlPath=/home/labex/.ansible/cp/ansible-ssh-%h-%p-%r 10.10.10.10 '/bin/sh -c '"'"'echo ~labex && sleep 0'"'"''

This level of detail can be invaluable for troubleshooting SSH connection issues.

Using the --limit Option

When working with a large inventory, you can limit Ansible to run commands against specific hosts or groups using the --limit option:

ansible-playbook test_playbook.yml --limit localhost

This command will only run the playbook against localhost, avoiding the 'UNREACHABLE!' error from virtual-host.

Now that we understand the 'UNREACHABLE!' error better, let's move on to troubleshooting and fixing these issues in the next step.

Troubleshooting and Fixing 'UNREACHABLE!' Errors

Now that we understand what causes 'UNREACHABLE!' errors, let's learn how to troubleshoot and fix them. We'll use a variety of approaches to diagnose and resolve connectivity issues.

Fixing Inventory Issues

One of the most common causes of 'UNREACHABLE!' errors is incorrect inventory information. Let's fix our inventory file:

cd ~/project/ansible-lab

First, let's update our inventory file to include a valid host. In this lab environment, we'll focus on using localhost with different connection methods to demonstrate troubleshooting techniques:

cat > inventory << 'EOF'
[local]
localhost ansible_connection=local

[ssh_local]
local-ssh ansible_host=127.0.0.1 ansible_connection=ssh

[virtual]
virtual-host ansible_host=10.10.10.10
EOF

We've added a new group ssh_local with a host that will attempt to connect to localhost via SSH instead of the local connection method.

Testing SSH Connectivity Directly

Before using Ansible, it's always a good idea to test SSH connectivity directly:

ssh 127.0.0.1

You might be prompted for a password or see a message about the host key. This is a good sign as it means SSH connectivity is working, but you might need to configure SSH properly for Ansible.

Press Ctrl+C to exit if you get stuck at the password prompt.

Setting Up SSH Keys for Passwordless Authentication

Ansible typically uses SSH keys for authentication. Let's set up passwordless SSH access to localhost:

## Generate an SSH key if you don't have one
ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa

## Add the key to authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

## Set proper permissions
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Now try connecting to localhost via SSH:

ssh 127.0.0.1

You should be able to connect without being prompted for a password. Type exit to return to your original session.

Testing Ansible with SSH Connection

Now, let's test Ansible with the SSH connection to localhost:

ansible ssh_local -m ping

If the SSH setup is correct, you should see a successful response:

local-ssh | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

If you still see an 'UNREACHABLE!' error, let's add more connection parameters to our inventory file:

cat > inventory << 'EOF'
[local]
localhost ansible_connection=local

[ssh_local]
local-ssh ansible_host=127.0.0.1 ansible_connection=ssh ansible_user=labex ansible_ssh_private_key_file=~/.ssh/id_rsa

[virtual]
virtual-host ansible_host=10.10.10.10
EOF

Try the ping command again:

ansible ssh_local -m ping

Using Ansible with a Custom SSH Configuration

Sometimes, you need more complex SSH configurations. Let's create a custom SSH config file:

mkdir -p ~/.ssh
cat > ~/.ssh/config << 'EOF'
Host local-ssh
    HostName 127.0.0.1
    User labex
    IdentityFile ~/.ssh/id_rsa
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null
EOF

## Set proper permissions
chmod 600 ~/.ssh/config

Update the inventory to use the SSH config entry:

cat > inventory << 'EOF'
[local]
localhost ansible_connection=local

[ssh_local]
local-ssh

[virtual]
virtual-host ansible_host=10.10.10.10
EOF

Test the connection again:

ansible ssh_local -m ping

Creating a Playbook to Test All Connections

Let's create a comprehensive playbook to test all our connections:

cat > connection_test.yml << 'EOF'
---
- name: Test Local Connection
  hosts: local
  gather_facts: no
  tasks:
    - name: Ping local
      ping:
      register: local_ping
    
    - name: Display local ping result
      debug:
        var: local_ping

- name: Test SSH Connection
  hosts: ssh_local
  gather_facts: no
  tasks:
    - name: Ping via SSH
      ping:
      register: ssh_ping
    
    - name: Display SSH ping result
      debug:
        var: ssh_ping
EOF

Run the playbook:

ansible-playbook connection_test.yml

You should see successful connections to both the local and SSH hosts:

PLAY [Test Local Connection] ********************************************

TASK [Ping local] ******************************************************
ok: [localhost]

TASK [Display local ping result] ****************************************
ok: [localhost] => {
    "local_ping": {
        "changed": false,
        "ping": "pong"
    }
}

PLAY [Test SSH Connection] **********************************************

TASK [Ping via SSH] ****************************************************
ok: [local-ssh]

TASK [Display SSH ping result] *****************************************
ok: [local-ssh] => {
    "ssh_ping": {
        "changed": false,
        "ping": "pong"
    }
}

PLAY RECAP *************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
local-ssh                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The successful output confirms that we've fixed the 'UNREACHABLE!' errors for our valid hosts. The only host that remains unreachable is virtual-host, which is intentional since it doesn't exist.

You have now successfully diagnosed and fixed 'UNREACHABLE!' errors by:

1. Testing direct SSH connectivity
2. Setting up SSH keys for passwordless authentication
3. Configuring Ansible inventory with proper connection parameters
4. Using a custom SSH configuration
5. Verifying connectivity with a comprehensive playbook

Implementing Best Practices to Prevent 'UNREACHABLE!' Errors

Now that we've fixed the immediate 'UNREACHABLE!' errors, let's focus on best practices to prevent them in the future. This involves proper inventory management, connection configurations, and error handling techniques.

Creating a Robust Inventory Structure

A well-organized inventory makes troubleshooting easier. Let's create a more structured inventory directory:

cd ~/project/ansible-lab
mkdir -p inventory/{group_vars,host_vars}

Now, let's create a main inventory file:

cat > inventory/hosts << 'EOF'
## Production Servers
[production]
## prod-server ansible_host=prod.example.com

## Development Servers
[development]
## dev-server ansible_host=dev.example.com

## Local Connections
[local]
localhost ansible_connection=local

## SSH Connections
[ssh_local]
local-ssh ansible_host=127.0.0.1
EOF

Next, let's create group variables for the SSH connection group:

cat > inventory/group_vars/ssh_local.yml << 'EOF'
---
ansible_connection: ssh
ansible_user: labex
ansible_ssh_private_key_file: ~/.ssh/id_rsa
ansible_ssh_common_args: '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
EOF

Update the Ansible configuration to use the new inventory directory:

cat > ansible.cfg << 'EOF'
[defaults]
inventory = ./inventory/hosts
host_key_checking = False
retry_files_enabled = True
retry_files_save_path = ~/.ansible/retry-files
timeout = 30
connect_timeout = 30
command_timeout = 30

[ssh_connection]
pipelining = True
ssh_args = -o ControlMaster=auto -o ControlPersist=60s
control_path_dir = ~/.ansible/cp
EOF

Creating a Connection Test Playbook with Retry Logic

Ansible allows you to retry failed tasks. Let's create a playbook with retry logic:

cat > connection_test_with_retry.yml << 'EOF'
---
- name: Test All Connections
  hosts: all
  gather_facts: no
  tasks:
    - name: Ping hosts
      ping:
      register: ping_result
      retries: 3
      delay: 5
      until: ping_result is not failed
      ignore_unreachable: yes
      
    - name: Display ping status
      debug:
        msg: "Connection to {{ inventory_hostname }} was successful"
      when: ping_result is success
      
    - name: Report unreachable hosts
      debug:
        msg: "Host {{ inventory_hostname }} is unreachable"
      when: ping_result is unreachable
EOF

Run the playbook with our new inventory structure:

ansible-playbook connection_test_with_retry.yml

You should see output showing successful connections to localhost and local-ssh.

Handling 'UNREACHABLE!' Errors Gracefully

Let's create a more advanced playbook that handles 'UNREACHABLE!' errors gracefully and generates a report:

cat > connection_report.yml << 'EOF'
---
- name: Test Connections and Generate Report
  hosts: all
  gather_facts: no
  tasks:
    - name: Try to connect to hosts
      ping:
      register: ping_result
      ignore_unreachable: yes
      
    - name: Create reachable hosts list
      set_fact:
        reachable_hosts: "{{ (reachable_hosts | default([])) + [inventory_hostname] }}"
      when: ping_result is success
      delegate_to: localhost
      delegate_facts: true
      
    - name: Create unreachable hosts list
      set_fact:
        unreachable_hosts: "{{ (unreachable_hosts | default([])) + [inventory_hostname] }}"
      when: ping_result is unreachable
      delegate_to: localhost
      delegate_facts: true

- name: Generate Connection Report
  hosts: localhost
  gather_facts: no
  tasks:
    - name: Display reachable hosts
      debug:
        msg: "Reachable hosts: {{ reachable_hosts | default([]) | join(', ') }}"
      
    - name: Display unreachable hosts
      debug:
        msg: "Unreachable hosts: {{ unreachable_hosts | default([]) | join(', ') }}"
      
    - name: Write report to file
      copy:
        content: |
          Connection Report
          -----------------
          Reachable hosts: {{ reachable_hosts | default([]) | join(', ') }}
          Unreachable hosts: {{ unreachable_hosts | default([]) | join(', ') }}
          
          Generated on: {{ ansible_date_time.iso8601 }}
        dest: ~/project/ansible-lab/connection_report.txt
      register: report
      
    - name: Show report location
      debug:
        msg: "Report saved to {{ report.dest }}"
EOF

Run the report playbook:

ansible-playbook connection_report.yml

Let's check the report:

cat ~/project/ansible-lab/connection_report.txt

You should see a report listing the reachable and unreachable hosts.

Using Ansible Inventory Plugins

Ansible provides inventory plugins to dynamically manage hosts. Let's create a simple script to demonstrate this:

cat > inventory_script.py << 'EOF'
#!/usr/bin/env python3

import json
import socket

def is_host_reachable(host, port=22, timeout=1):
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(timeout)
        result = sock.connect_ex((host, port))
        sock.close()
        return result == 0
    except:
        return False

## Define our hosts
hosts = {
    'localhost': '127.0.0.1',
    'local-ssh': '127.0.0.1',
    'virtual-host': '10.10.10.10'
}

## Check reachability and build inventory
inventory = {
    'all': {
        'hosts': list(hosts.keys())
    },
    'reachable': {
        'hosts': []
    },
    'unreachable': {
        'hosts': []
    },
    '_meta': {
        'hostvars': {}
    }
}

for hostname, ip in hosts.items():
    reachable = is_host_reachable(ip)
    group = 'reachable' if reachable else 'unreachable'
    inventory[group]['hosts'].append(hostname)
    
    inventory['_meta']['hostvars'][hostname] = {
        'ansible_host': ip,
        'reachability_checked': True,
        'is_reachable': reachable
    }

print(json.dumps(inventory, indent=2))
EOF

chmod +x inventory_script.py

Test the dynamic inventory script:

./inventory_script.py

You should see JSON output showing the hosts categorized as reachable or unreachable.

Let's run a playbook using this dynamic inventory:

ansible-playbook -i ./inventory_script.py connection_test.yml --limit reachable

This will only attempt to connect to hosts that the script has determined are reachable, helping you avoid 'UNREACHABLE!' errors altogether.

These best practices provide a robust framework for managing Ansible connectivity and preventing 'UNREACHABLE!' errors in production environments.

Summary

In this lab, you learned how to identify, troubleshoot, and prevent the 'UNREACHABLE!' error in Ansible. You have:

1. Set up a basic Ansible environment and encountered the 'UNREACHABLE!' error firsthand

2. Analyzed the error message and understood common causes of connectivity issues

3. Used various troubleshooting techniques to diagnose connection problems

4. Implemented solutions to fix the errors, including:

   • Configuring SSH keys for passwordless authentication
   • Setting up proper inventory files
   • Using SSH configuration options

5. Applied best practices to prevent future 'UNREACHABLE!' errors, such as:

   • Creating a structured inventory organization
   • Implementing retry logic
   • Developing error handling strategies
   • Using dynamic inventory scripts to check host reachability

These skills will help you maintain reliable Ansible deployments and quickly resolve any connectivity issues that arise. By understanding the underlying causes of 'UNREACHABLE!' errors and implementing proper preventive measures, you can ensure your infrastructure automation runs smoothly and efficiently.