Introduction
Linux logs are the cornerstone of system monitoring and troubleshooting, providing invaluable insights into the inner workings of your system. In this tutorial, we will explore the fundamental concepts of Linux logs and how to effectively monitor them using the versatile 'tail' command.
You will learn where log files are stored, how to view their contents, track real-time changes, and extract specific information. These skills are essential for anyone working with Linux systems, allowing you to diagnose problems, monitor system health, and understand what your system is doing behind the scenes.
Understanding Linux Log Files and Their Locations
Linux systems maintain detailed records of various activities, errors, and events in specialized text files called log files. These logs serve as the main source of information when troubleshooting issues or monitoring system health.
Where are Linux Logs Stored?
Most Linux distributions, including Ubuntu 22.04, store log files in the /var/log directory. Let's explore this directory to see what log files are available on your system.
Open your terminal and enter the following command:
ls -l /var/log
You should see output similar to this:
total 1048
drwxr-x--- 2 root adm 4096 May 2 09:52 apache2
drwxr-xr-x 2 root root 4096 Apr 30 15:27 apt
-rw-r----- 1 syslog adm 183867 May 2 10:18 auth.log
-rw-r--r-- 1 root root 1112 Nov 17 2022 alternatives.log
drwxr-xr-x 2 root root 4096 Apr 20 2022 cups
-rw-r----- 1 syslog adm 308342 May 2 10:18 kern.log
-rw-rw-r-- 1 root utmp 0 Apr 20 2022 lastlog
drwxr-xr-x 2 root root 4096 Nov 8 2022 nginx
-rw-r----- 1 syslog adm 473079 May 2 10:18 syslog
The exact files and directories will vary depending on what software is installed on your system.
Common Log Files and Their Contents
Here are some of the most important log files you'll encounter:
/var/log/syslog: Contains general system messages/var/log/auth.log: Records authentication attempts and user management/var/log/kern.log: Contains kernel messages and errors/var/log/apt: Directory containing package management logs
Let's examine the contents of one of these files. Since we're learning about log monitoring, we'll start with a basic view of the system log:
sudo head -n 20 /var/log/syslog
The head command displays the first few lines of a file (20 in this case). We use sudo because some log files can only be read by users with administrative privileges.
You should see output similar to:
May 2 08:23:15 ubuntu systemd[1]: Started Daily apt download activities.
May 2 08:23:15 ubuntu systemd[1]: Starting Daily apt upgrade and clean activities...
May 2 08:23:15 ubuntu systemd[1]: apt-daily-upgrade.service: Deactivated successfully.
May 2 08:23:15 ubuntu systemd[1]: Finished Daily apt upgrade and clean activities.
May 2 08:37:16 ubuntu systemd[1]: Starting Message of the Day...
May 2 08:37:16 ubuntu systemd[1]: motd-news.service: Deactivated successfully.
May 2 08:37:16 ubuntu systemd[1]: Finished Message of the Day.
...
Notice the format of each log entry:
- Date and time
- Hostname (ubuntu in this example)
- Program or service generating the log
- The actual log message
Understanding Log Entry Structure
Most log entries follow a standard format, making them easier to read and process. The typical format includes:
- Timestamp: When the event occurred
- Hostname: The name of the machine
- Service/Application: What generated the log
- Message: The actual information about what happened
Now that you understand where logs are located and how they're structured, let's move on to viewing the latest entries in a log file using the tail command.
Using the Basic 'tail' Command to View Log Files
While the head command shows the beginning of a file, the tail command shows the end of a file. This is particularly useful for log files, as the most recent entries are usually at the end.
Basic Usage of the 'tail' Command
The basic syntax of the tail command is:
tail [options] [file]
By default, tail displays the last 10 lines of a file. Let's try it with the system log:
sudo tail /var/log/syslog
You should see the 10 most recent log entries:
May 2 10:15:32 ubuntu systemd[1]: Starting Clean php session files...
May 2 10:15:32 ubuntu systemd[1]: phpsessionclean.service: Deactivated successfully.
May 2 10:15:32 ubuntu systemd[1]: Finished Clean php session files.
May 2 10:17:01 ubuntu CRON[8752]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
May 2 10:18:14 ubuntu sshd[8755]: Accepted password for labex from 192.168.1.5 port 52413 ssh2
May 2 10:18:14 ubuntu sshd[8755]: pam_unix(sshd:session): session opened for user labex(uid=1000) by (uid=0)
May 2 10:18:14 ubuntu systemd-logind[698]: New session 4 of user labex.
May 2 10:18:14 ubuntu systemd[1]: Started Session 4 of User labex.
May 2 10:18:15 ubuntu sshd[8789]: Received disconnect from 192.168.1.5 port 52413:11: disconnected by user
May 2 10:18:15 ubuntu sshd[8789]: Disconnected from user labex 192.168.1.5 port 52413
Viewing a Custom Number of Lines
Sometimes 10 lines aren't enough to get the context you need. You can specify how many lines to display using the -n option:
sudo tail -n 5 /var/log/syslog
This will display only the last 5 lines of the file:
May 2 10:18:14 ubuntu sshd[8755]: pam_unix(sshd:session): session opened for user labex(uid=1000) by (uid=0)
May 2 10:18:14 ubuntu systemd-logind[698]: New session 4 of user labex.
May 2 10:18:14 ubuntu systemd[1]: Started Session 4 of User labex.
May 2 10:18:15 ubuntu sshd[8789]: Received disconnect from 192.168.1.5 port 52413:11: disconnected by user
May 2 10:18:15 ubuntu sshd[8789]: Disconnected from user labex 192.168.1.5 port 52413
Viewing Authentication Logs
The /var/log/auth.log file contains information about user authentication and security-related events. Let's look at the most recent entries:
sudo tail -n 15 /var/log/auth.log
You might see output similar to:
May 2 10:18:14 ubuntu sshd[8755]: Accepted password for labex from 192.168.1.5 port 52413 ssh2
May 2 10:18:14 ubuntu sshd[8755]: pam_unix(sshd:session): session opened for user labex(uid=1000) by (uid=0)
May 2 10:18:14 ubuntu systemd-logind[698]: New session 4 of user labex.
May 2 10:18:15 ubuntu sshd[8789]: Received disconnect from 192.168.1.5 port 52413:11: disconnected by user
May 2 10:18:15 ubuntu sshd[8789]: Disconnected from user labex 192.168.1.5 port 52413
May 2 10:18:15 ubuntu sshd[8755]: pam_unix(sshd:session): session closed for user labex
May 2 10:18:15 ubuntu systemd-logind[698]: Session 4 logged out. Waiting for processes to exit.
May 2 10:18:15 ubuntu systemd-logind[698]: Removed session 4.
May 2 10:20:01 ubuntu sudo: labex : TTY=pts/0 ; PWD=/home/labex ; USER=root ; COMMAND=/usr/bin/tail /var/log/syslog
May 2 10:20:01 ubuntu sudo: pam_unix(sudo:session): session opened for user root(uid=0) by labex(uid=1000)
May 2 10:20:01 ubuntu sudo: pam_unix(sudo:session): session closed for user root
May 2 10:21:15 ubuntu sudo: labex : TTY=pts/0 ; PWD=/home/labex ; USER=root ; COMMAND=/usr/bin/tail -n 5 /var/log/syslog
May 2 10:21:15 ubuntu sudo: pam_unix(sudo:session): session opened for user root(uid=0) by labex(uid=1000)
May 2 10:21:15 ubuntu sudo: pam_unix(sudo:session): session closed for user root
This log shows user login attempts, session openings and closings, and sudo command usage. It's an essential log for security monitoring and troubleshooting user access issues.
Looking at Kernel Logs
The kernel is the core of the Linux operating system. The /var/log/kern.log file contains messages generated by the kernel, which are useful for diagnosing hardware and driver issues:
sudo tail -n 10 /var/log/kern.log
The output might look like:
May 2 09:15:02 ubuntu kernel: [16789.456123] USB 1-1: new high-speed USB device number 3 using xhci_hcd
May 2 09:15:02 ubuntu kernel: [16789.605432] usb 1-1: New USB device found, idVendor=0781, idProduct=5571, bcdDevice= 1.00
May 2 09:15:02 ubuntu kernel: [16789.605436] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
May 2 09:15:02 ubuntu kernel: [16789.605438] usb 1-1: Product: Ultra
May 2 09:15:02 ubuntu kernel: [16789.605440] usb 1-1: Manufacturer: SanDisk
May 2 09:15:02 ubuntu kernel: [16789.605442] usb 1-1: SerialNumber: 4C530001211124120222
May 2 09:15:02 ubuntu kernel: [16789.606521] usb-storage 1-1:1.0: USB Mass Storage device detected
May 2 09:15:02 ubuntu kernel: [16789.606705] scsi host3: usb-storage 1-1:1.0
May 2 09:15:03 ubuntu kernel: [16790.624553] scsi 3:0:0:0: Direct-Access SanDisk Ultra 1.00 PQ: 0 ANSI: 6
May 2 09:15:03 ubuntu kernel: [16790.625045] sd 3:0:0:0: Attached scsi generic sg1 type 0
Now that you understand how to view the latest entries in log files, let's move on to monitoring logs in real-time, which is one of the most powerful features of the tail command.
Real-time Log Monitoring with 'tail -f'
One of the most powerful features of the tail command is its ability to monitor files in real-time using the -f option (short for "follow"). This is particularly useful for observing log files as new entries are being added, which helps with troubleshooting and monitoring.
Understanding 'tail -f'
The -f option tells tail to keep the file open and continuously display new lines as they are added to the file. This creates a real-time view of the log file, allowing you to see events as they happen.
Monitoring System Logs in Real-time
Let's use the -f option to monitor the system log in real-time:
sudo tail -f /var/log/syslog
After running this command, you'll see the most recent entries in the log file, and the terminal will remain open, showing new entries as they appear.
To generate some log entries, let's open a new terminal window (press Ctrl+Alt+T to open a new terminal) and run a few basic commands:
logger "This is a test message from user $(whoami)"
The logger command sends a message to the system log. Now, go back to your first terminal window where you're running tail -f, and you should see your test message appear in the log:
May 2 10:34:45 ubuntu labex: This is a test message from user labex
Try running the logger command a few more times with different messages to see them appear in real-time:
logger "System test number 2"
logger "Monitoring logs is fun!"
Each of these messages should appear in your tail -f output window as they are logged.
When you're done watching the log, press Ctrl+C to stop the tail -f command and return to the command prompt.
Monitoring Authentication Activity
Let's monitor the authentication log to observe login and authentication-related events:
sudo tail -f /var/log/auth.log
This will show you real-time authentication activities, such as user logins, sudo command usage, and authentication failures.
To generate some auth log entries, open a new terminal and run a sudo command:
sudo ls /root
You should see new entries appear in your tail -f window showing the sudo activity:
May 2 10:38:23 ubuntu sudo: labex : TTY=pts/1 ; PWD=/home/labex ; USER=root ; COMMAND=/usr/bin/ls /root
May 2 10:38:23 ubuntu sudo: pam_unix(sudo:session): session opened for user root(uid=0) by labex(uid=1000)
May 2 10:38:23 ubuntu sudo: pam_unix(sudo:session): session closed for user root
Again, press Ctrl+C to stop the tail -f command when you're done observing.
Practical Use Cases for Real-time Log Monitoring
Real-time log monitoring is incredibly useful in several scenarios:
- Troubleshooting issues: When something isn't working, watching logs in real-time can help identify the problem.
- Security monitoring: Observing authentication logs can help detect unauthorized access attempts.
- Application debugging: Developers can watch application logs to understand what's happening during testing.
- System administration: Administrators can monitor system logs during maintenance or configuration changes.
For example, if you were troubleshooting a network issue, you might monitor the system log while trying to connect to see any error messages that appear.
By mastering real-time log monitoring with tail -f, you gain a powerful tool for understanding your system's behavior as it happens.
Filtering Log Information for Better Analysis
Log files can be very large and contain a lot of information. To make log analysis more manageable, you can combine the tail command with other tools like grep to filter and find specific information.
Using 'grep' with 'tail'
The grep command searches for patterns in text. When combined with tail, it allows you to filter log entries for specific keywords or patterns. The basic syntax is:
tail [options] [file] | grep "pattern"
Let's look at some practical examples:
Finding Error Messages
To find error messages in the system log, you can filter for the word "error":
sudo tail -n 100 /var/log/syslog | grep -i "error"
The -i option with grep makes the search case-insensitive, so it will match "error", "Error", "ERROR", etc. You might see output like:
May 2 09:23:45 ubuntu systemd[1]: Failed to start Apache HTTP Server: Job failed with error code.
May 2 09:45:12 ubuntu kernel: [12345.678901] Buffer I/O error on device sdb1, logical block 123456
If you don't see any results, it means there are no error messages in the last 100 lines of the log.
Monitoring Specific Services
You can also filter logs to monitor a specific service or program. For example, to monitor SSH-related activity:
sudo tail -n 100 /var/log/auth.log | grep "sshd"
This will show only the SSH-related entries in the authentication log:
May 2 10:18:14 ubuntu sshd[8755]: Accepted password for labex from 192.168.1.5 port 52413 ssh2
May 2 10:18:14 ubuntu sshd[8755]: pam_unix(sshd:session): session opened for user labex(uid=1000) by (uid=0)
May 2 10:18:15 ubuntu sshd[8789]: Received disconnect from 192.168.1.5 port 52413:11: disconnected by user
May 2 10:18:15 ubuntu sshd[8789]: Disconnected from user labex 192.168.1.5 port 52413
May 2 10:18:15 ubuntu sshd[8755]: pam_unix(sshd:session): session closed for user labex
Real-time Filtered Monitoring
You can also combine real-time monitoring with filtering to watch for specific events as they happen:
sudo tail -f /var/log/syslog | grep --line-buffered "test"
The --line-buffered option ensures that grep outputs each matching line immediately, rather than buffering them, which is important when using tail -f.
Now, in another terminal, generate a test log message:
logger "This is a test message for grep filtering"
In your first terminal, you should only see this message appear (and not any other log messages) because of the grep filter for "test".
Advanced Filtering with Multiple Patterns
You can also search for multiple patterns using the OR operator (|) in grep:
sudo tail -n 100 /var/log/syslog | grep -E "error|warning|critical"
The -E option enables extended regular expressions, allowing you to use the OR operator. This command will show lines containing any of the words "error", "warning", or "critical".
Extracting Specific Information with 'awk'
For more advanced filtering and formatting, you can use the awk command. For example, to extract just the time and message from syslog:
sudo tail -n 10 /var/log/syslog | awk '{print $3, $5, $6, $7, $8, $9, $10}'
This extracts specific fields from each log entry, making it easier to focus on the most relevant information.
By mastering these filtering techniques, you can efficiently analyze large log files and extract exactly the information you need for troubleshooting and monitoring.
Summary
Congratulations on completing this lab on monitoring real-time changes in Linux log files with the 'tail' command. You have learned several important skills:
Understanding Linux log files: You now know where logs are stored, what they contain, and how log entries are structured.
Basic log viewing: You can use the
tailcommand to view the most recent entries in log files and specify how many lines to display.Real-time monitoring: With
tail -f, you can watch logs as they update, allowing you to observe system activities and troubleshoot issues as they occur.Filtering log information: By combining
tailwith tools likegrepandawk, you can extract specific information from logs, making analysis more efficient.
These skills are essential for anyone working with Linux systems, from beginners to experienced administrators. Log analysis is a fundamental part of system maintenance, troubleshooting, and security monitoring.
As you continue your Linux journey, you'll find that these log monitoring techniques will help you better understand your system, diagnose problems more quickly, and maintain a more secure and reliable environment.
