Kali Log Analysis with journalctl

Introduction

In this lab, you will learn the essentials of log analysis in a Kali Linux environment using the powerful journalctl tool and other command-line utilities. Designed for beginners, this hands-on experience focuses on fundamental skills for system administration and debugging. You will explore how to access, view, and analyze system logs within a Kali Linux container on a LabEx VM.

Through step-by-step practical exercises, you will view system logs stored in directories like /var/log, filter logs by specific services, search for errors, and export logs for further analysis. All operations will be performed within the Kali Linux container's shell, which you will automatically enter when opening the terminal. This lab provides a controlled environment to build your skills in monitoring and troubleshooting system events.

Skills Graph

Viewing System Logs in Kali Linux

In this first step, you will learn how to view system logs within a Kali Linux container on the LabEx VM. System logs are critical for monitoring and troubleshooting issues on a Linux system, as they record events, errors, and activities. As a beginner, understanding how to access these logs is a foundational skill for system administration.

When you open the terminal in the LabEx VM environment, you will be automatically connected to the Kali Linux container's shell. There is no need to manually start the container or enter the shell; the environment is already set up for you. All operations in this lab will be performed within this container's terminal.

Linux systems store logs in directories like /var/log. Common log files include /var/log/syslog for general system messages and /var/log/auth.log for authentication-related events. Let's start by viewing the contents of these files using simple commands.

Run the following command in the terminal to display the last 10 lines of the /var/log/syslog file:

tail -n 10 /var/log/syslog

This command uses tail to show the most recent entries in the log file. The output will look something like this, though the exact content depends on your system's activity:

Oct 15 10:22:34 kali-container kernel: [ 1234.567890] some kernel message
Oct 15 10:23:01 kali-container cron[1234]: (CRON) INFO (pidfile fd = 3)
...

Each line typically includes a timestamp, hostname, service or process name, and the event details. This output confirms that you have successfully accessed the system logs.

To continuously monitor new log entries in real-time, which is useful for debugging, run this command:

tail -f /var/log/syslog

This will display new entries as they are added. To stop monitoring, press Ctrl + C to return to the terminal prompt.

Next, let's check the authentication log file /var/log/auth.log, which records login attempts and other security events. Run this command:

tail -n 5 /var/log/auth.log

The output might look like this, showing authentication-related activities:

Oct 15 10:20:01 kali-container sudo: root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/bin/bash
...

This step introduces you to viewing logs, a basic but essential skill. In the following steps, you will build on this by filtering and analyzing these logs for specific information.

Installing and Using journalctl for Log Viewing

Now that you have seen how to view logs using basic commands like tail, let's move to a more powerful tool called journalctl. This tool is used to query and display logs managed by systemd, the system and service manager in modern Linux distributions, including Kali Linux. As a beginner, learning journalctl is important because it provides a structured way to access detailed system logs.

Since you are already in the Kali Linux container's shell (automatically entered when opening the terminal), you don't need to navigate or start anything manually. However, journalctl might not be preinstalled or fully functional in a minimal container setup, so we will ensure the necessary tools are available by updating the package list and installing required components.

Run the following commands to update the package list and install systemd components if needed:

apt update
apt install -y systemd

These commands refresh the package repository and install systemd, which includes journalctl. Wait for the installation to complete; it may take a few moments.

Once installed, you can use journalctl to view system logs. Run this command to display the most recent log entries:

journalctl -n 10

The -n 10 option limits the output to the last 10 lines. The output will look similar to this, though it depends on your system's activity:

Oct 15 10:25:01 kali-container systemd[1]: Starting system activity...
Oct 15 10:25:02 kali-container systemd[1]: Started system activity...
...

This output shows log entries with timestamps, service names, and messages. If you see something like this, it means journalctl is working correctly. If no output appears or you encounter an error, it might be due to limited log data in the container environment, which is normal for this lab setup.

To view logs in real-time, similar to tail -f, run this command:

journalctl -f

This will continuously display new log entries. Press Ctrl + C to stop monitoring and return to the terminal prompt.

This step helps you understand how to use journalctl for viewing logs, which is more advanced than basic file viewing. In the next step, you will learn to filter these logs for specific services.

Filtering Logs by Service with journalctl

Building on your ability to view logs with journalctl, this step focuses on filtering logs to display entries related to a specific service. As a beginner, filtering is a key skill because system logs often contain a mix of messages from various processes, and isolating relevant information helps in troubleshooting specific issues.

You are still working within the Kali Linux container's shell, which you automatically entered when opening the terminal. No manual navigation or container startup is needed. We will use journalctl to filter logs for a particular service, such as cron, which manages scheduled tasks in Linux. This allows you to focus on logs related to a single component without being overwhelmed by unrelated data.

To filter logs for the cron service, run the following command:

journalctl -u cron

The -u option specifies the unit (service) to filter by, in this case, cron. The output will show logs specifically related to the cron service, looking something like this, depending on system activity:

Oct 15 10:30:01 kali-container cron[1234]: (CRON) INFO (pidfile fd = 3)
Oct 15 10:30:02 kali-container cron[1235]: (CRON) STARTUP (fork ok)
...

This output displays only cron-related entries, including timestamps and event details. If no output appears, it might mean the cron service hasn't generated recent logs in this container, which is acceptable for learning purposes.

If cron logs are not available or you want to explore another service, you can try filtering for sshd (if installed), which handles SSH connections. First, ensure openssh-server is installed by running:

apt install -y openssh-server

Wait for the installation to complete. Then, filter logs for sshd with this command:

journalctl -u ssh

The output might show SSH-related events if the service is active. If no output appears, it’s fine for this lab as the focus is on learning the filtering process.

This step teaches you how to narrow down log data to specific services using journalctl. In the next step, you will learn to search for errors within these logs to identify potential issues.

Searching Logs for Errors with journalctl

Having learned to filter logs by service, this step will guide you through searching system logs for errors using journalctl. As a beginner, identifying errors in logs is crucial for troubleshooting, as logs often contain messages indicating problems with services or applications. This skill helps you quickly pinpoint issues that need attention.

You are still operating within the Kali Linux container's shell, automatically accessed when you open the terminal. No additional setup or navigation is required. We will use journalctl to search for error messages by filtering logs based on priority levels, where errors are typically logged at a specific severity.

In journalctl, log entries are categorized by priority levels, with 3 representing errors (critical issues) and 4 representing warnings. To search for error messages across all logs, run this command:

journalctl -p 3

The -p 3 option filters logs to show only entries at priority level 3 (errors). The output might look like this, depending on your system's activity:

Oct 15 10:35:10 kali-container some-service[5678]: error: failed to initialize module
...

This output displays error messages with timestamps and details. If no output appears, it means there are no recent errors at this priority level in the logs, which is common in a controlled container environment.

To include warnings along with errors, you can search for priority levels 3 and 4 by running:

journalctl -p 3..4

This command shows both errors and warnings. The output might include additional lines like:

Oct 15 10:36:15 kali-container another-service[7890]: warning: connection timeout
...

If no output appears, it’s acceptable for this lab as the goal is to learn the search process. This step equips you with the ability to identify critical issues in logs using journalctl. In the next step, you will learn to export these logs for further analysis.

Exporting Logs for Analysis with journalctl

In this step, you will learn how to export system logs for further analysis using journalctl. Building on the previous steps where you viewed, filtered, and searched logs, exporting allows you to save specific log data to a file. This is useful for detailed examination or sharing with others for troubleshooting. As a beginner, this skill helps you document findings without repeatedly accessing the original logs.

You are still working within the Kali Linux container's shell, which you automatically entered when opening the terminal. No manual container startup or navigation is needed. We will use journalctl to extract logs and redirect the output to files in the /root directory for easy access.

Let's export all recent logs to a file named system_logs.txt in the /root directory. Run this command:

journalctl -n 50 > /root/system_logs.txt

The -n 50 option limits the output to the last 50 log entries, and the > symbol redirects the output to the specified file. You won't see the output in the terminal because it is being saved to the file. To confirm the file was created and contains data, run:

cat /root/system_logs.txt

The output will show the saved log entries, similar to what you saw with journalctl -n 50, including timestamps and messages.

Next, let's export logs filtered by a specific service, such as cron, to a file named cron_logs.txt. Run this command:

journalctl -u cron > /root/cron_logs.txt

This saves only cron-related logs to /root/cron_logs.txt. Verify the content by running:

cat /root/cron_logs.txt

The output will display cron service logs if available, confirming the export was successful.

Finally, let's export error logs to a file named error_logs.txt. Run this command:

journalctl -p 3 > /root/error_logs.txt

This saves error messages to the specified file. Check the content with:

cat /root/error_logs.txt

This output will show error logs if any exist. By completing this step, you have learned to save log data for later review, a practical skill for system administration and debugging.

Summary

In this lab, you have learned the essentials of log analysis in Kali Linux using the journalctl tool and basic command-line utilities. You started by viewing system logs in the /var/log directory with commands like tail, then progressed to using journalctl for more structured log access. Through hands-on steps, you filtered logs by service, searched for errors, and exported logs for further analysis within a Kali Linux container on the LabEx VM. These foundational skills equip you for effective system administration and debugging in Linux environments.