Linux Log Analysis with Journalctl for Incident Response

Introduction

In the fields of system administration and cybersecurity, log analysis is a critical skill. System logs record a wide range of events, from routine operations to critical errors and potential security breaches. Being able to effectively navigate and interpret these logs is essential for monitoring system health, troubleshooting problems, and responding to security incidents.

This lab introduces you to journalctl, the standard tool for querying and displaying logs from the journald service on modern Linux systems. You will learn how to perform basic log analysis tasks that form the foundation of monitoring and incident response.

Throughout this lab, you will:

Review system boot logs.
Filter logs to find specific events like authentication failures.
Simulate and detect a suspicious event.
Export logs for further analysis.

Review System Boot Logs with Journalctl

In this step, you will learn how to use the journalctl command to review system logs, specifically focusing on the messages generated during the most recent boot process. This is a common first step when diagnosing startup issues.

The journalctl command allows you to query the contents of the systemd journal. Without any arguments, it displays all logs, which can be overwhelming.

To make the output more manageable, we can use the -b or --boot flag to view only the logs from the current boot session.

Execute the following command in your terminal to view the logs for the current boot:

journalctl -b

You will see a paged output starting with the earliest messages from the boot process. You can use the Up and Down arrow keys to navigate. Press q to exit the pager and return to the command prompt.

-- Journal begins at Tue 2023-10-31 08:30:00 UTC, ends at Tue 2023-10-31 09:00:00 UTC. --
Oct 31 08:30:01 labex-vm kernel: Linux version 5.15.0-87-generic ...
Oct 31 08:30:01 labex-vm kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-87-generic ...
Oct 31 08:30:01 labex-vm kernel: KERNEL supported cpus:
Oct 31 08:30:01 labex-vm kernel:   Intel GenuineIntel
Oct 31 08:30:01 labex-vm kernel:   AMD AuthenticAMD
...
(END)

This command is invaluable for understanding what services started successfully and for identifying any errors that occurred during system startup.

Filter Logs for Authentication Failures

In this step, you will filter the journal to find specific events, such as failed authentication attempts, which are critical for security monitoring. A common target for attackers is the SSH service, so monitoring its logs is a high priority.

We can use the -u flag with journalctl to filter logs by a specific systemd unit. For the SSH service, the unit is typically ssh.service (on Ubuntu/Debian) or sshd.service (on Red Hat/CentOS).

Let's filter the logs to see only the entries related to the SSH daemon. Note that you may need to use sudo to view system logs:

sudo journalctl -u ssh

This command shows all log entries generated by the sshd service. To narrow down our search to potential security issues, we can pipe this output to the grep command to search for keywords like "Failed".

Run the following command to find failed password attempts for the SSH service:

sudo journalctl -u ssh | grep "Failed password"

If there have been no recent failed login attempts, this command may produce no output. This is normal on a secure system. In the next step, we will generate such an event ourselves to see how it appears in the logs.

Simulate a Suspicious Event and Analyze Logs

Now, let's simulate a suspicious event and then use our log analysis skills to detect it. A common sign of a brute-force attack is a series of failed login attempts. We will simulate one such attempt by trying to SSH into our own machine (localhost) with a username that does not exist.

Execute the following command in your terminal. You will be prompted for a password; you can enter anything as it is expected to fail.

ssh non_existent_user@localhost

The system will deny the connection, which is the expected outcome. You should see a message like this:

non_existent_user@localhost's password:
Permission denied, please try again.
non_existent_user@localhost's password:
Permission denied (publickey,password).

Now that we have generated a failed login event, let's re-run our log analysis command from the previous step to see if we can find it.

sudo journalctl -u ssh | grep "Failed password"

This time, the command should produce output showing the failed attempt we just made.

Oct 31 09:15:12 labex-vm sshd[1234]: Failed password for invalid user non_existent_user from 127.0.0.1 port 48492 ssh2

This simple exercise demonstrates the core loop of incident response: an event occurs, it gets logged, and an administrator or automated system analyzes the logs to detect it.

Export Logs for Centralized Analysis Simulation

In a real-world scenario, you would often export logs from individual machines to a centralized logging server (like a SIEM) for long-term storage and correlation. In this step, we'll simulate this by exporting our recent SSH logs to a file.

journalctl can output logs in various formats. The json-pretty format is particularly useful as it's both human-readable and easily parsed by other tools.

Let's export all SSH logs from the last 10 minutes into a file named ssh_logs.json in your current directory (~/project).

sudo journalctl -u ssh --since "10 minutes ago" -o json-pretty > ~/project/ssh_logs.json

Now, verify that the file has been created:

ls -l ~/project

You should see ssh_logs.json in the file listing.

total 4
-rw-r--r-- 1 labex labex 1234 Oct 31 09:20 ssh_logs.json

Finally, let's view the contents of our exported log file.

cat ~/project/ssh_logs.json

The output will be a structured JSON array, with each log entry as an object. This format is ideal for ingestion into other analysis platforms.

[
    {
        "__CURSOR" : "s=...",
        "__REALTIME_TIMESTAMP" : "...",
        "__MONOTONIC_TIMESTAMP" : "...",
        "_BOOT_ID" : "...",
        "_TRANSPORT" : "syslog",
        "PRIORITY" : "6",
        "SYSLOG_FACILITY" : "4",
        "SYSLOG_IDENTIFIER" : "sshd",
        "MESSAGE" : "Failed password for invalid user non_existent_user from 127.0.0.1 port 48492 ssh2",
        "_PID" : "1234",
        ...
    }
]

You have successfully simulated the process of preparing logs for centralized analysis.

Summary

In this lab, you gained hands-on experience with journalctl, a powerful tool for log analysis on modern Linux systems. These are fundamental skills for any system administrator, DevOps engineer, or security professional.

You have learned how to:

Review system and boot logs to diagnose startup issues.
Filter logs by specific services and message content to find relevant events.
Identify suspicious activity, such as failed logins, from log entries.
Export logs in a structured format like JSON for centralized storage and further analysis.

Mastering these techniques will enable you to better monitor your systems, troubleshoot issues more efficiently, and take the first steps in responding to security incidents.

Monitoring and Incident Response Log Analysis