Analyze Logs in Red Hat Enterprise Linux

Introduction

In this lab, you will gain hands-on experience with analyzing and storing system logs on Red Hat Enterprise Linux 9 using journalctl and rsyslog. You will begin by understanding the core architecture of system logging, including the roles of systemd-journald and rsyslog, and identifying key log files. Subsequently, you will learn to review and filter syslog files using common commands, manually send custom syslog messages, and explore and filter system journal entries with journalctl. The lab also covers configuring persistent system journal storage and maintaining accurate system time using timedatectl and chronyd, equipping you with essential skills for system analysis and troubleshooting.

Understand System Log Architecture and Key Files

In this step, you will learn about the fundamental components of system logging in Red Hat Enterprise Linux 9, specifically focusing on the systemd-journald and rsyslog services, and the key log files they manage. Understanding this architecture is crucial for effective system analysis and troubleshooting.

First, let's explore the systemd-journald service. This service is at the core of the operating system's event logging. It collects event messages from various sources, including the system kernel, early boot processes, standard output/error from daemons, and syslog events. The systemd-journald service stores these logs in a structured, indexed binary file called the journal.

Next, we'll look at the rsyslog service. While systemd-journald collects logs, rsyslog reads syslog messages from the journal as they arrive. It then processes these messages and records them to traditional log files in the /var/log directory or forwards them to other services based on its configuration. These rsyslog files persist across reboots.

Let's examine some of the important log files managed by rsyslog in the /var/log directory. You can use the ls command to list the contents of this directory.

ls /var/log

You will see a list of various log files. Based on your system, you should see files like anaconda, audit, boot.log, chrony, cron, dnf.log, messages, secure, and others. Some of the most common ones include:

• /var/log/messages: This file contains most general syslog messages, excluding those related to authentication, email processing, scheduled job execution, and debugging.
• /var/log/secure: This file stores syslog messages related to security and authentication events.
• /var/log/maillog: This file contains syslog messages about the mail server.
• /var/log/cron: This file logs syslog messages about scheduled job execution.
• /var/log/boot.log: This file contains non-syslog console messages about system startup.

To view the contents of these files, you can use commands like cat, less, or tail. Note that most log files in /var/log require root privileges to read, so you'll need to use sudo. For example, let's view the last few lines of the /var/log/messages file using tail.

sudo tail /var/log/messages

You can also view the contents of the /var/log/secure file, which is important for security auditing.

sudo tail /var/log/secure

Understanding the purpose of these files will help you quickly locate relevant information when troubleshooting system issues.

Review and Filter Syslog Files with common commands

In this step, you will learn how to effectively review and filter syslog files using common Linux commands. Syslog messages are categorized by facility (which subsystem produces the message) and priority (the message's severity). Understanding these categories is crucial for efficient log analysis.

First, let's review the concept of Syslog Facilities and Priorities.

Syslog Facilities: These indicate the source of the log message. Examples include kern (kernel messages), user (user-level messages), mail (mail system messages), daemon (system daemons messages), auth (authentication and security messages), and cron (clock daemon messages).

Syslog Priorities: These define the severity of the message, ranging from emerg (system unusable) to debug (debugging-level message). The order of severity from highest to lowest is: emerg, alert, crit, err, warning, notice, info, debug.

Log files can grow very large, making it difficult to find specific information. Therefore, filtering is essential. You can use commands like grep, awk, and sed to filter log file content.

Let's start by viewing the entire content of /var/log/messages using less. This command allows you to scroll through the file. Press q to exit less. Remember to use sudo since log files require root privileges to read.

sudo less /var/log/messages

Now, let's try to filter messages. Suppose you are interested in messages related to authentication. These messages are typically found in /var/log/secure. Use grep to search for lines containing the word "authentication" in /var/log/secure. Remember to use sudo to access the log file.

sudo grep "authentication" /var/log/secure

You might not see any output if there are no recent authentication messages. Let's try a more common search term, like "sshd", which relates to the SSH daemon.

sudo grep "sshd" /var/log/secure

You should see output showing SSH daemon activities, authentication attempts, or other security-related events. The exact output will depend on the current system activity and may include entries like:

Dec 16 10:15:30 host sshd[1234]: Accepted publickey for labex from 172.25.0.10 port 12345 ssh2
Dec 16 10:15:30 host sshd[1234]: pam_unix(sshd:session): session opened for user labex(uid=1000) by (uid=0)
...output omitted...

Log messages also contain timestamps. You can filter messages by date and time. For instance, to see messages from a specific date, you can combine grep with the date. Let's try to find messages from today's date in /var/log/messages. Use the current date format that appears in your system logs.

sudo grep "$(date '+%b %d')" /var/log/messages

Log File Rotation:
To prevent log files from consuming too much disk space, systems use logrotate. This utility rotates log files, renaming old ones (e.g., /var/log/messages becomes /var/log/messages-20220320) and creating new, empty ones. After a certain period (typically four weeks), the oldest rotated log files are discarded. A scheduled job runs logrotate daily to manage this process.

You can observe rotated log files by listing the contents of /var/log again. You might see files with date extensions or .gz extensions (if they have been compressed).

ls -l /var/log/messages*

Example output:

-rw-------. 1 root root 123456 Mar 20 23:59 /var/log/messages
-rw-------. 1 root root  78901 Mar 13 23:59 /var/log/messages-20220320
-rw-------. 1 root root  54321 Mar 06 23:59 /var/log/messages-20220313.gz
...output omitted...

This shows how logrotate manages older log files.

Finally, let's analyze the anatomy of a syslog entry. A typical log message in /var/log/secure looks like this:

Dec 16 10:11:48 host sshd[1433]: Failed password for student from 172.25.0.10 port 59344 ssh2

• Dec 16 10:11:48: The timestamp of the log entry.
• host: The host that sent the log message.
• sshd[1433]: The program or process name (sshd) and its PID (1433) that sent the log message.
• Failed password for …: The actual message content.

Understanding this format helps you parse and interpret log entries more effectively.

Send Custom Syslog Messages Manually

In this step, you will learn how to send custom syslog messages manually using the logger command. This is a useful technique for testing rsyslog service configurations or for injecting custom messages into the system logs for debugging or monitoring purposes.

The logger command sends messages to the rsyslog service. By default, it sends the message with the user facility and notice priority (user.notice), unless specified otherwise with the -p option.

Let's start by sending a simple test message to the default log location, which is typically /var/log/messages.

logger "This is a test message from labex."

After executing the command, you can verify that the message was logged by checking the /var/log/messages file. Use tail to view the last few lines of the file. Remember to use sudo to access the log file.

sudo tail /var/log/messages

You should see your custom message at the end of the output, similar to this:

...
Dec 16 10:30:00 host labex: This is a test message from labex.

Now, let's try sending a message with a specific facility and priority. Recall from the previous step that syslog messages are categorized by facility and priority. For example, local7.notice means the message will be logged with the local7 facility and notice priority. The local7 facility is often used for custom applications or boot messages, and it's typically directed to /var/log/boot.log by rsyslog configuration.

To send a message to the rsyslog service to be recorded in the /var/log/boot.log log file, execute the following logger command:

logger -p local7.notice "Log entry created by labex for boot.log"

Now, verify that this message has been written to /var/log/boot.log. Use sudo to access the log file.

sudo tail /var/log/boot.log

You should see your custom message in the output:

...
Dec 16 10:31:00 host labex: Log entry created by labex for boot.log

This demonstrates how you can control where your custom messages are logged by specifying the facility and priority. This capability is very useful for testing rsyslog configurations and for injecting specific events into your system logs.

Explore and Filter System Journal Entries with journalctl

In this step, you will learn how to explore and filter system journal entries using the journalctl command. As you learned, the systemd-journald service stores logging data in a structured, indexed binary file called the journal. The journalctl command is your primary tool for interacting with this journal.

Let's start by viewing all messages in the journal. When you run journalctl without any options, it displays all available log entries, starting from the oldest. Since you are logged in as labex with sudo privileges, you will have full access to the journal.

journalctl

You will see a large amount of output. Press q to exit the journalctl viewer. Notice that journalctl highlights important log messages: messages at notice or warning priority are in bold text, while messages at the error priority or higher are in red text.

Now, let's explore some common journalctl options for filtering and viewing specific entries.

1. View the last N log entries (-n option):
By default, journalctl -n shows the last 10 log entries. You can specify a different number, for example, the last 5 entries:

journalctl -n 5

You should see the 5 most recent log entries.

2. Follow new journal entries (-f option):
Similar to the tail -f command, the journalctl -f option outputs the last 10 lines of the system journal and continues to output new journal entries as they are appended. This is very useful for real-time monitoring.

journalctl -f

To exit this continuous output, press Ctrl+C.

3. Filter by priority (-p option):
You can filter the output by the priority level of the journal entries. The journalctl -p option shows entries at a specified priority level (by name or by number) or higher. The priority levels, in ascending order, are debug, info, notice, warning, err, crit, alert, and emerg.

Let's list journal entries at the err priority or higher:

journalctl -p err

You might see error messages related to various system components.

4. Filter by systemd unit (-u option):
You can show messages for a specified systemd unit by using the journalctl -u option and the unit name. For example, to view logs specifically for the sshd service:

journalctl -u sshd.service

This will display all log entries related to the SSH daemon.

5. Filter by time range (--since and --until options):
When looking for specific events, you can limit the output to a specific time frame. Both --since and --until options take a time argument in the "YYYY-MM-DD hh:mm:ss" format. Double quotation marks are required if there are spaces in the argument. You can also use relative terms like yesterday, today, tomorrow, or time durations like "-1 hour".

Let's view all journal entries from the beginning of today:

journalctl --since today

Now, let's view entries from the last hour:

journalctl --since "-1 hour"

6. View verbose output (-o verbose option):
To see additional details about each log entry, including internal journal fields, you can use the -o verbose option. This can be helpful for advanced troubleshooting.

journalctl -n 1 -o verbose

This will show the last log entry with all its verbose details. Notice fields like _COMM (command name), _EXE (path to executable), _PID (process ID), _UID (user ID), and _SYSTEMD_UNIT (systemd unit). These fields can be used for more precise filtering.

For example, to find entries from a specific sshd process with a known PID (you can get a PID from a previous journalctl -u sshd.service output):

journalctl _SYSTEMD_UNIT=sshd.service _PID=<PID_NUMBER>

Replace <PID_NUMBER> with an actual PID you observed from sshd entries. For instance, if you saw sshd[1433], you would use _PID=1433.

journalctl _SYSTEMD_UNIT=sshd.service _PID=1433

This command demonstrates how you can combine multiple filters to narrow down your search in the journal.

Configure Persistent System Journal Storage

In this step, you will learn how to configure the system journal to persist across reboots. By default, Red Hat Enterprise Linux 9 stores the system journal in the /run/log directory, which is a temporary file system. This means that all journal entries are cleared after a system reboot. To retain historical log data, you need to configure the systemd-journald service for persistent storage.

The configuration for systemd-journald is located in the /etc/systemd/journald.conf file. The Storage parameter within this file determines whether journals are stored in a volatile manner or persistently.

The Storage parameter can be set to one of the following values:

• persistent: Stores journals in the /var/log/journal directory, which persists across reboots. If this directory does not exist, systemd-journald will create it.
• volatile: Stores journals in the temporary /run/log/journal directory. Data in /run does not persist across reboots. This is the default behavior if Storage is not explicitly set and /var/log/journal does not exist.
• auto: If the /var/log/journal directory exists, systemd-journald uses persistent storage; otherwise, it uses volatile storage. This is the default if you do not set the Storage parameter.
• none: No storage is used. All logs are dropped, though they can still be forwarded.

Let's modify the /etc/systemd/journald.conf file to enable persistent journal storage. You will use sudo nano to edit this file.

sudo nano /etc/systemd/journald.conf

Inside the nano editor, find the [Journal] section. Uncomment the Storage line (remove the # at the beginning) and set its value to persistent.

Your file should look similar to this (only showing relevant lines):

[Journal]
Storage=persistent
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#Audit=no
#FlushIntervalSec=
#SyncIntervalSec=
#ReadKMsg=yes
#Audit=yes

After making the change, save the file by pressing Ctrl+X, then Y to confirm saving, and Enter to confirm the filename.

For the changes to take effect, you need to restart the systemd-journald service. Since this environment is a Docker container, systemctl is not available. Instead, we will simulate the effect of restarting the service by ensuring the /var/log/journal directory is created and has the correct permissions, which systemd-journald would do upon restart in a non-containerized environment.

First, let's create the directory if it doesn't exist and set appropriate permissions.

sudo mkdir -p /var/log/journal
sudo chown root:systemd-journal /var/log/journal
sudo chmod 2755 /var/log/journal

Now, to verify that the persistent storage is configured and active, you can check if the /var/log/journal directory contains subdirectories with hexadecimal names and .journal files. These files store the structured and indexed journal entries.

ls -l /var/log/journal

You should see a subdirectory with a long hexadecimal name (e.g., 4ec03abd2f7b40118b1b357f479b3112). Inside this directory, you will find .journal files.

ls -l /var/log/journal/ <YOUR_HEX_ID> /

Replace <YOUR_HEX_ID> with the actual hexadecimal ID you found in the previous ls command output. For example:

ls -l /var/log/journal/4ec03abd2f7b40118b1b357f479b3112/

You should see files like system.journal and possibly user-1000.journal.

Even with persistent journals, the system does not keep all data forever. The journal has a built-in log rotation mechanism. You can configure size limits and retention periods in the /etc/systemd/journald.conf file using parameters like SystemMaxUse, SystemKeepFree, SystemMaxFileSize, and MaxRetentionSec.

Finally, when persistent journals are enabled, journalctl output includes entries from the current system boot as well as previous system boots. To limit the output to a specific system boot, use the journalctl -b option.

To list all recognized system boot events:

journalctl --list-boots

You will see a list of boot IDs and their corresponding time ranges. The current boot is usually 0. Previous boots are negative numbers (-1, -2, etc.).

To view entries from the current system boot only:

journalctl -b

To view entries from the previous boot (e.g., the one before the current one, usually -1):

journalctl -b -1

This concludes the configuration of persistent journal storage.

Maintain Accurate System Time with timedatectl and chronyd

In this step, you will learn how to maintain accurate system time using the timedatectl command and understand the role of the chronyd service. Accurate timekeeping is crucial for logging, security, and many network services.

1. Using timedatectl to manage system time and time zones:

The timedatectl command provides an overview of the current time-related system settings, including the local time, universal time (UTC), RTC time, time zone, and NTP synchronization status.

Let's check the current time settings of your system:

timedatectl

You should see output similar to this (the exact time and date will reflect your current system time):

               Local time: Sun 2025-06-15 21:46:11 EDT
           Universal time: Mon 2025-06-16 01:46:11 UTC
                 RTC time: Mon 2025-06-16 01:46:10
                Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

You can list all available time zones using the list-timezones option:

timedatectl list-timezones | less

Press q to exit less. The time zones are named based on the Internet Assigned Numbers Authority (IANA) time zone database, typically by continent/ocean and then the largest city.

To change the system's time zone, you use the set-timezone option. For example, let's change the time zone to America/Phoenix. You need sudo privileges for this.

sudo timedatectl set-timezone America/Phoenix

Now, verify the change:

timedatectl

You should see the time zone updated to America/Phoenix.

You can also manually set the system's current time using the set-time option. The format is "YYYY-MM-DD hh:mm:ss", but you can omit the date or time. Let's set the time to 09:00:00 (for the current date).

sudo timedatectl set-time 09:00:00

Verify the time change:

timedatectl

Finally, the set-ntp option enables or disables NTP synchronization for automatic time adjustment. It takes true or false as an argument. Let's disable NTP synchronization for a moment (we will re-enable it later).

sudo timedatectl set-ntp false

Verify the NTP service status:

timedatectl

You should see NTP service: inactive.

2. Understanding and configuring the chronyd service:

The chronyd service is a daemon that keeps the system's Real-Time Clock (RTC) accurate by synchronizing it with Network Time Protocol (NTP) servers. It's the default NTP client in Red Hat Enterprise Linux.

The configuration file for chronyd is /etc/chrony.conf. By default, it uses public NTP servers. In a real-world scenario, you might configure it to use internal NTP servers.

Let's view the default chrony.conf file.

cat /etc/chrony.conf

You will see lines starting with server or pool, which define the NTP sources. The iburst option is recommended as it takes four measurements quickly for more accurate initial synchronization.

The stratum of an NTP time source indicates its quality. A stratum 0 is a reference clock, stratum 1 is directly attached to a reference clock, and stratum 2 synchronizes from a stratum 1 server.

Since systemctl is not available in this container environment, we cannot directly restart chronyd to apply configuration changes. However, we can simulate the configuration change by modifying the file.

Let's re-enable NTP synchronization using timedatectl.

sudo timedatectl set-ntp true

Verify the NTP service status again:

timedatectl

You should see NTP service: active.

The chronyc command acts as a client to the chronyd service. You can use it to monitor the synchronization status. The chronyc sources command shows the current time sources and their synchronization status.

chronyc sources -v

The output will show details about the NTP sources. The asterisk * in the S (Source state) field indicates the source that chronyd is currently synchronized to.

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
 / .- Source state '*' = current best, '+' = combined, '-' = not combined,
| /             'x' = may be in error, '~' = too variable, '?' = unusable.
||                                                 .- xxxx [ yyyy ] +/- zzzz
||      Reachability register (octal) -.           |  xxxx = adjusted offset,
||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
||                                \     |          |  zzzz = estimated error.
||                                 |    |           \
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^* 100.100.61.88                 1   5   377    16  +1824us[+2180us] +/-   85ms
...output omitted...

This output confirms that your system is actively synchronizing its time with an NTP server.

Summary

In this lab, we gained a comprehensive understanding of system log architecture in Red Hat Enterprise Linux 9, focusing on the interplay between systemd-journald and rsyslog. We learned that systemd-journald collects and stores structured, indexed binary logs in the journal, while rsyslog processes these messages and writes them to traditional log files in /var/log. We explored key log files like /var/log/messages, /var/log/secure, and others, and practiced reviewing and filtering syslog files using common commands. We also learned how to manually send custom syslog messages.

Furthermore, we delved into exploring and filtering system journal entries using journalctl, understanding its capabilities for accessing detailed system events. We then configured persistent system journal storage to ensure log data retention across reboots. Finally, we covered maintaining accurate system time using timedatectl and chronyd, recognizing its importance for accurate log timestamps and overall system integrity. This lab provided essential skills for effective system analysis and troubleshooting through robust log management.