Popular Searches

System Log Analysis

LinuxBeginner
Practice Now

Introduction

System logs are the primary source of truth when investigating security incidents or troubleshooting system errors in Linux. They record almost everything that happens on a server, from user authentications to kernel messages.

Logs matter because most attacks leave traces before they leave obvious damage. Failed logins, unusual sudo usage, service crashes, configuration changes, and strange process activity often show up in logs first. A strong analyst learns to treat logs not as background noise, but as a timeline that explains what the system experienced.

In this lab, you will step into the shoes of a system administrator. You will learn how to analyze standard Linux system logs using powerful command-line text processing tools like grep and awk. Specifically, you will parse authentication logs to detect simulated SSH brute-force login attempts and track commands executed with elevated sudo privileges.

Locate System Logs

In this step, you will familiarize yourself with the location and structure of Linux system logs.

Before searching for threats, it helps to understand what kind of evidence each log source is meant to hold. auth.log focuses on identity and privilege events. syslog is broader and mixes messages from many services. Knowing the role of each file helps you choose the right place to look instead of grepping blindly through everything.

Typically, Linux system logs are stored in the /var/log directory. The two most important log files for security analysis are:

  • auth.log: Records all authentication-related events, such as SSH logins and sudo usage.
  • syslog: Records general system activity and application messages.

Because reading actual system logs often requires root privileges and the logs can be overwhelmingly large, we have prepared sample auth.log and syslog files in your current working directory (~/project).

First, let's verify that the log files are present in your directory. Use the ls command to list the files:

ls -l

You should see an output similar to this:

total 8
-rw-r--r-- 1 labex labex 584 Oct 10 10:30 auth.log
-rw-r--r-- 1 labex labex 245 Oct 10 10:30 syslog

Next, let's peek at the contents of the authentication log. The head command is useful for viewing the first few lines of a file without flooding your terminal screen.

Execute the following command to view the beginning of auth.log:

head auth.log

You will see several lines of text. Each line represents a single log entry, typically starting with a timestamp, followed by the hostname, the service that generated the log (like sshd or sudo), and the specific message.

Oct 10 10:00:01 labex sshd[1234]: Failed password for invalid user admin from 192.168.1.100 port 33452 ssh2
Oct 10 10:05:12 labex sshd[1235]: Failed password for root from 10.0.0.5 port 44212 ssh2
...

Analyze auth.log for Failed Logins

In this step, you will use the grep command to filter the log file and identify potential security threats, such as failed SSH login attempts.

Failed logins are not automatically malicious, but repeated failures are an important weak signal. A user may mistype a password once. A large pattern of failures from one source or against many accounts suggests guessing activity. Good log analysis often starts by turning weak individual events into stronger patterns through grouping and counting.

grep is a powerful command-line utility used to search for specific text patterns within files. When an attacker tries to guess passwords (a brute-force attack), the SSH service (sshd) logs a "Failed password" message for every unsuccessful attempt.

Let's search the auth.log file for all instances of failed logins. Run the following command:

grep "Failed password" auth.log

Your terminal will output only the lines that contain the exact phrase "Failed password":

Oct 10 10:00:01 labex sshd[1234]: Failed password for invalid user admin from 192.168.1.100 port 33452 ssh2
Oct 10 10:05:12 labex sshd[1235]: Failed password for root from 10.0.0.5 port 44212 ssh2
Oct 10 10:20:01 labex sshd[1240]: Failed password for invalid user test from 172.16.0.4 port 12345 ssh2

If you are analyzing a real server, there might be thousands of these lines. To quickly count how many failed attempts occurred, you can pipe (|) the output of grep into the wc -l command. The wc command stands for "word count", and the -l flag tells it to count lines.

Execute the following command to count the failed logins:

grep "Failed password" auth.log | wc -l

You should see the number 3 printed to your terminal, indicating there were three failed login attempts in our sample log.

Track Sudo Privilege Escalation

In this step, you will learn how to track commands executed with elevated privileges. When a user runs a command using sudo, it is recorded in the auth.log file.

First, let's use grep to find all sudo activities in the log:

grep "sudo:" auth.log

You will see output detailing who used sudo and what command they ran:

Oct 10 10:15:22 labex sudo:    labex : TTY=pts/0 ; PWD=/home/labex/project ; USER=root ; COMMAND=/usr/bin/cat /etc/shadow
Oct 10 10:25:33 labex sudo:    labex : TTY=pts/0 ; PWD=/home/labex/project ; USER=root ; COMMAND=/usr/bin/apt update

While grep is great for filtering lines, sometimes you only want to extract a specific piece of information from those lines, such as the exact command that was executed. This is where awk comes in.

This is an important progression for beginners: first find relevant lines, then extract the fields that answer your question. Security work often alternates between these two operations. Filtering narrows the evidence set. Field extraction turns that evidence into something you can summarize or compare.

awk is a versatile programming language designed for text processing and data extraction. By default, awk splits lines into "fields" based on spaces. However, you can specify a custom delimiter using the -F flag.

Notice that in the sudo log entries, the actual command executed comes right after the string COMMAND=. We can tell awk to split the line using COMMAND= as the separator, and then print the second part of the split line.

Run the following command to extract just the executed commands:

grep "sudo:" auth.log | awk -F 'COMMAND=' '{print $2}'

Let's break down how this works:

  • grep "sudo:" auth.log filters the log for sudo entries.
  • | passes those entries to awk.
  • -F 'COMMAND=' tells awk to use COMMAND= as the dividing line.
  • '{print $2}' tells awk to print the second piece of the divided text (everything after COMMAND=).

You should see a clean list of the executed commands:

/usr/bin/cat /etc/shadow
/usr/bin/apt update

Extract and Sort IP Addresses

In this final step, you will combine grep and awk with other command-line utilities to build a powerful data processing pipeline. Chaining commands together is a common practice for system administrators to quickly extract useful intelligence from raw logs without writing complex scripts.

Pipelines are powerful because they let you build investigations incrementally. One command selects the events, another extracts the field you care about, and another sorts or counts the results. This way of thinking scales from quick incident triage to larger automation later in the course.

We want to extract just the IP addresses of attackers who failed to log in, sort them, and remove any duplicates.

First, let's look at a clever awk trick. In our failed login logs, the IP address is always the 4th word from the end of the line, regardless of whether the username was valid or invalid. awk has a built-in variable called NF (Number of Fields), which represents the total number of words in a line. Therefore, $(NF-3) will always point to the IP address!

Let's construct our pipeline step-by-step:

  • grep finds the failed attempts.
  • awk '{print $(NF-3)}' extracts the IP address.
  • sort organizes the IP addresses numerically.
  • uniq removes any duplicate IP addresses so each attacker is only listed once.

Run the complete pipeline and redirect the final output to a file named suspicious_ips.txt:

grep "Failed password" auth.log | awk '{print $(NF-3)}' | sort | uniq > suspicious_ips.txt

Now, view the contents of your newly created file using the cat command:

cat suspicious_ips.txt

You should see the extracted, unique IP addresses printed to your terminal:

10.0.0.5
172.16.0.4
192.168.1.100

Summary

Congratulations! In this lab, you have successfully learned how to perform basic system log analysis in Linux.

You have mastered the following skills:

  • Locating and viewing standard Linux system logs.
  • Using grep to filter logs for specific security events, such as failed SSH logins.
  • Utilizing awk to parse log lines and extract specific fields, like commands executed via sudo.
  • Combining these tools into a command-line pipeline to extract and sort malicious IP addresses.

These text processing skills are fundamental for any system administrator, DevOps engineer, or cybersecurity professional working in a Linux environment.