Identifying Malware Indicators on Linux

Introduction

In this lab, you will learn fundamental techniques for identifying potential malware indicators on a Linux system. Through a series of hands-on exercises, you will use standard command-line utilities to investigate common signs of a compromise, gaining practical skills in system monitoring and basic forensic analysis.

You will begin by observing suspicious process activity, learning how to spot unusual CPU and memory spikes that can signal a malicious process. Next, you will focus on the file system to detect unexpected file creations and modifications, a key indicator of unauthorized activity. Finally, you will practice analyzing system logs to find anomalous entries that could point to a security breach.

Observe Suspicious Process Activity for CPU and Memory Spikes

In this step, you will learn how to use standard Linux utilities to monitor system processes for unusual spikes in CPU and memory usage. A sudden, high consumption of resources is a common indicator of malicious software (malware), a compromised process, or a misbehaving application. We will use the top command for real-time monitoring and the stress tool to safely simulate a high-load scenario. All commands will be run from your ~/project directory.

First, you need to install the stress utility, as it is not included in the base system. We'll use the apt-get package manager. The first command updates your package list, and the second installs stress.

sudo apt-get update
sudo apt-get install stress -y

Once the installation is complete, let's establish a baseline by viewing the current processes on the system. The top command provides a real-time, dynamic view of the running system.

top

You will see a full-screen interface that continuously updates. Pay attention to the columns %CPU (CPU usage) and %MEM (memory usage). Under normal conditions, these values should be relatively low for most processes. Press the q key to exit top and return to your command prompt.

top - 16:20:42 up 10 min,  1 user,  load average: 0.00, 0.01, 0.00
Tasks: 100 total,   1 running,  99 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.1 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   1987.2 total,    845.5 free,    450.1 used,    691.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   1355.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0  169404  13120   8484 S   0.0   0.6   0:01.55 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
...

Now, let's simulate a suspicious process that consumes a high amount of CPU. We'll use stress to start one worker that will perform CPU-intensive calculations for 60 seconds. The & at the end runs the command in the background, allowing you to continue using the terminal.

stress --cpu 1 --timeout 60 &

You will see a process ID (PID) for the background job. Now, quickly run top again to observe the effect.

top

This time, you should see the stress process at or near the top of the list, with its %CPU value close to 100.0. This is exactly what you would look for when hunting for a process that is hogging the CPU. After 60 seconds, the stress process will automatically terminate. Press q to exit top.

Next, let's simulate a process that consumes a large amount of memory. This command will start one worker that allocates 256 Megabytes of memory and holds it for 60 seconds.

stress --vm 1 --vm-bytes 256M --timeout 60 &

Again, run top immediately to see the impact.

top

In the top output, find the stress process. This time, its %CPU will be low, but its %MEM value will be significantly higher than other processes, demonstrating a memory leak or a memory-intensive attack. This technique helps you identify anomalous resource consumption that warrants further investigation. Press q to exit.

Detect Unexpected File Creations and Modifications

In this step, you will learn how to detect suspicious file system activity, such as the unexpected creation or modification of files. Attackers often create or alter files to establish persistence, store malicious payloads, or tamper with system configurations. We will use two powerful command-line tools: find to search for files based on their modification times, and inotifywait to perform real-time monitoring of directory events. All operations will take place in your ~/project directory.

First, let's use the find command to locate files that have been recently modified. This is useful for periodic security audits. Let's create a sample configuration file to monitor.

echo "SERVER_IP=192.168.1.1" > ~/project/app.conf

The find command can search for files based on various criteria. We'll use the -mmin flag, which stands for "modified minutes ago". The following command will find all files in the current directory (.) that were modified within the last 5 minutes.

find . -mmin -5

The output will list the file you just created.

./app.conf

Now, let's simulate an attacker modifying this configuration file.

echo "MALICIOUS_PAYLOAD=..." >> ~/project/app.conf

If you run the find command again, it will once again show app.conf because its modification time has been updated. This is a simple way to spot recent changes. Now, let's simulate an attacker dropping a new file, a common tactic.

touch ~/project/hidden_script.sh

Run the find command one more time.

find . -mmin -5

Now it lists both the modified file and the newly created one, demonstrating how you can quickly identify recent file system activity.

./app.conf
./hidden_script.sh

While find is useful for audits, it doesn't provide real-time alerts. For that, we can use inotify-tools. First, you must install the package.

sudo apt-get update
sudo apt-get install -y inotify-tools

The core command is inotifywait, which can monitor a file or directory for specific events. Let's run it in monitor mode (-m) on our ~/project directory. The & will run it as a background process so you can continue to use your terminal.

inotifywait -m ~/project &

The command is now silently watching for changes. Let's perform a few actions to trigger it. First, create a file.

echo "new file" > ~/project/new_file.txt

You will immediately see output from inotifywait reporting the CREATE event.

/home/labex/project/ CREATE new_file.txt

Now, modify the file.

echo "modified" >> ~/project/new_file.txt

The tool reports a MODIFY event.

/home/labex/project/ MODIFY new_file.txt

Finally, delete the file.

rm ~/project/new_file.txt

You will see a DELETE event. This real-time feedback is invaluable for a Security Operations Center (SOC) or an automated intrusion detection system.

/home/labex/project/ DELETE new_file.txt

To stop the background monitoring process, you can use the pkill command.

pkill inotifywait

You have successfully used both find and inotifywait to detect file system changes.

Analyze System Logs for Anomalous Entries

In this step, you will learn how to analyze system logs to find evidence of suspicious activity. System logs are records of events that occur on the system, and they are a critical source of information for security monitoring and forensic analysis. We will focus on authentication logs, which track user logins and privilege escalations, and use standard Linux tools like grep, tail, and journalctl to find anomalous entries.

Most system logs on a Linux system are stored in the /var/log directory. One of the most important for security is /var/log/auth.log, which records authentication-related events, including SSH logins and sudo command usage. Let's start by viewing the last few lines of this file using the tail command. You need sudo because this file is protected.

sudo tail /var/log/auth.log

You will see a series of timestamped entries. The format typically includes the date, time, hostname, the process that generated the log, and the event message.

Jul 22 10:30:01 labex-vm sudo:    labex : TTY=pts/0 ; PWD=/home/labex/project ; USER=root ; COMMAND=/usr/bin/tail /var/log/auth.log
Jul 22 10:30:01 labex-vm sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Jul 22 10:30:01 labex-vm sudo: pam_unix(sudo:session): session closed for user root

Now, let's simulate a common suspicious event: a failed login attempt. Attackers often try to guess passwords for valid or common usernames. We can simulate this by trying to SSH into our own machine (localhost) with a username that doesn't exist, like baduser.

ssh baduser@localhost

The system will prompt you for a password (you can use labex as the password). Since the user doesn't exist, any password will fail. Simply press Ctrl+C to cancel the password prompt and return to the terminal. The connection attempt will fail, but it will be logged.

Now, let's hunt for the evidence. We can use the grep command to filter the auth.log file for lines containing the word "failed". This quickly isolates unsuccessful events.

sudo grep "Failed" /var/log/auth.log

The output will now clearly show the log entry for our failed SSH login attempt, which is a strong indicator of a potential intrusion attempt.

Jul 22 10:35:15 labex-vm sshd[12345]: Failed password for invalid user baduser from 127.0.0.1 port 54321 ssh2

Modern Linux systems using systemd also have a centralized logging system managed by journald. The journalctl command is a powerful tool to query these logs. Let's use it to find the same failed login event by searching for messages from the sshd process. Note that you must use sudo to view system-level logs.

sudo journalctl | grep sshd

Scroll through the output (using the arrow keys, and press q to exit) and you will find the same "Failed password" entry. To make it even easier, you can filter by message priority. Let's ask journalctl to show only entries with a priority of "error" (err) or higher for the sshd service.

sudo journalctl -p err | grep sshd

This command gives you a much more concise view of only the error conditions, making it very efficient for spotting problems.

-- Logs begin at ... --
Jul 22 10:35:15 labex-vm sshd[12345]: Failed password for invalid user baduser from 127.0.0.1 port 54321 ssh2
-- End of logs --

Finally, you can also manually inject messages into the logs using the logger command. This is useful for custom scripts to report security events. Let's create a custom alert.

logger "SECURITY-ALERT: Unauthorized file access detected in /etc"

Now, search for this specific alert in the system logs.

sudo grep "SECURITY-ALERT" /var/log/syslog

You will see your custom message, demonstrating how you can integrate your own monitoring with the system's logging facility.

Jul 22 10:40:00 labex-vm labex: SECURITY-ALERT: Unauthorized file access detected in /etc

You have now learned the basics of navigating system logs to find evidence of anomalous and potentially malicious activity.

Summary

In this lab, you learned essential techniques for identifying malware indicators on a Linux system. You practiced how to observe suspicious process activity by using the top command to monitor for abnormal CPU and memory spikes. To understand what this looks like in a real-world scenario, you used the stress utility to simulate a high-load process and compared its resource consumption against a normal system baseline.

The lab also covered how to detect unexpected file creations and modifications, which are common signs of a compromised system. Finally, you learned to analyze system logs for anomalous entries, a crucial skill for finding evidence of unauthorized access, misconfigurations, or other security-related events.