How to check if kernel performance counters are active in Linux

Introduction

In this lab, you will learn how to check if kernel performance counters are active in Linux. You will explore how to use the perf stat command to measure performance counters for command execution, verify the performance counter configuration within the /sys/kernel filesystem, and inspect kernel logs using dmesg for messages related to performance counters. This hands-on experience will provide you with the skills to diagnose and understand the status of performance monitoring on your Linux system.

Skills Graph

%%%%{init: {'theme':'neutral'}}%%%% flowchart RL linux(("Linux")) -.-> linux/TextProcessingGroup(["Text Processing"]) linux(("Linux")) -.-> linux/BasicSystemCommandsGroup(["Basic System Commands"]) linux(("Linux")) -.-> linux/BasicFileOperationsGroup(["Basic File Operations"]) linux(("Linux")) -.-> linux/FileandDirectoryManagementGroup(["File and Directory Management"]) linux/BasicSystemCommandsGroup -.-> linux/sleep("Execution Delaying") linux/BasicFileOperationsGroup -.-> linux/ls("Content Listing") linux/BasicFileOperationsGroup -.-> linux/cat("File Concatenating") linux/FileandDirectoryManagementGroup -.-> linux/cd("Directory Changing") linux/TextProcessingGroup -.-> linux/grep("Pattern Searching") subgraph Lab Skills linux/sleep -.-> lab-558797{{"How to check if kernel performance counters are active in Linux"}} linux/ls -.-> lab-558797{{"How to check if kernel performance counters are active in Linux"}} linux/cat -.-> lab-558797{{"How to check if kernel performance counters are active in Linux"}} linux/cd -.-> lab-558797{{"How to check if kernel performance counters are active in Linux"}} linux/grep -.-> lab-558797{{"How to check if kernel performance counters are active in Linux"}} end

Check counters with perf stat

In this step, you will learn how to use the perf stat command to measure performance counters. Performance counters are special hardware registers that count specific events happening within the CPU, such as the number of instructions executed, cache misses, or branch predictions.

The perf tool is a powerful command-line utility for performance analysis in Linux. perf stat is used to run a command and gather performance counter statistics for that command's execution.

Let's start by running a simple command and observing its performance counters. We will use the ls command, which lists directory contents.

Open the terminal if you haven't already. You can find the Xfce Terminal icon on the left side of your desktop.

Now, type the following command and press Enter:

perf stat ls

You should see output similar to this:

 Performance counter stats for 'ls':

          <placeholder>      task-clock (msec)         ##    <placeholder> CPUs utilized
          <placeholder>      context-switches          ##    <placeholder> /sec
          <placeholder>      cpu-migrations            ##    <placeholder> /sec
          <placeholder>      page-faults               ##    <placeholder> /sec
          <placeholder>      cycles                    ##    <placeholder> GHz
          <placeholder>      instructions              ##    <placeholder>  insns per cycle
          <placeholder>      branches                  ##    <placeholder> % of all instructions
          <placeholder>      branch-misses             ##    <placeholder> % of all branches

          <placeholder> msec task-clock                ##    <placeholder> CPUs utilized

The exact numbers and events might vary depending on the system and the specific execution, but you should see a list of performance events and their counts during the execution of the ls command.

Some common events you might see include:

task-clock: The total time the task was running on a CPU.
cycles: The number of CPU cycles spent.
instructions: The number of instructions executed.
branch-misses: The number of times a branch prediction was incorrect.

Understanding these counters can help you identify performance bottlenecks in your code or system.

You can also run perf stat on a more complex command or even a script to analyze its performance.

For example, let's try running perf stat on a simple sleep command:

perf stat sleep 1

This will measure the performance counters while the system sleeps for 1 second. The output will show the counters for the sleep process.

Experiment with running perf stat on different commands to see how the counters change.

Click Continue to proceed to the next step.

Verify counter config in /sys/kernel

In this step, you will explore how performance counter configurations are exposed in the Linux kernel's /sys filesystem. The /sys filesystem provides an interface to kernel data structures, allowing you to inspect and sometimes modify kernel parameters.

Performance counter settings, such as whether unprivileged users can access raw performance counter events, are often controlled through files in /sys/kernel/perf_event.

Let's navigate to the relevant directory and inspect a configuration file.

First, use the cd command to change your current directory to /sys/kernel/perf_event. Remember, your default directory is ~/project, so you need to use the absolute path.

Type the following command and press Enter:

cd /sys/kernel/perf_event

Now you are in the /sys/kernel/perf_event directory. You can use the ls command to see the files within this directory:

ls

You should see a list of files, which might include:

kptr_restrict  perf_event_paranoid  sysfs_deprecated

The file we are interested in is perf_event_paranoid. This file controls the level of access unprivileged users have to performance events. A lower value means less paranoia (more access), and a higher value means more paranoia (less access).

Let's view the content of this file using the cat command:

cat perf_event_paranoid

The output will be a single number, typically between -1 and 2.

<number>

Here's what the different values generally mean:

-1: Allow all users to use all perf features.
0: Allow all users to use perf for per-process and per-CPU measurements.
1: Allow only privileged users (like root) to use perf for per-process and per-CPU measurements.
2: Allow only privileged users to use perf for per-process measurements.

The default value is often 2 for security reasons, preventing unprivileged users from potentially using performance counters to gain information about other processes.

Understanding this file helps you see how the system is configured regarding performance monitoring access.

Click Continue to move on.

Inspect counter logs in dmesg

In this step, you will learn how to use the dmesg command to view kernel messages, which can sometimes include information related to performance counters or perf events. dmesg is a command that displays the kernel ring buffer messages. These messages are generated by the kernel during boot-up and runtime and can contain valuable information for debugging and system analysis.

While perf itself usually outputs statistics directly to the terminal, the kernel's interaction with performance monitoring hardware or any related issues might be logged in the kernel ring buffer.

To view the kernel messages, use the dmesg command:

dmesg

This will print a potentially large amount of output to your terminal, showing all the messages from the kernel ring buffer.

To find messages specifically related to perf or performance counters, you can pipe the output of dmesg to the grep command. grep is a powerful tool for searching text using patterns.

Let's search for messages containing the word "perf":

dmesg | grep perf

This command does the following:

dmesg: Outputs the kernel messages.
|: This is a pipe, which sends the output of the command on the left as input to the command on the right.
grep perf: Searches the input for lines containing the word "perf".

You might see output similar to this (or nothing, depending on recent kernel events):

[<placeholder>] perf: interrupt took too long (<placeholder> > <placeholder> ms), lowering kernel.perf_event_paranoid to <placeholder>

This example message indicates that a performance interrupt took too long, and the kernel automatically adjusted the perf_event_paranoid setting (which we saw in the previous step) to a less restrictive value to potentially mitigate the issue.

You can also search for other related terms, like "performance" or "counter":

dmesg | grep performance

dmesg | grep counter

Inspecting dmesg output can be helpful for diagnosing problems related to performance monitoring or understanding how the kernel is interacting with the performance counter hardware.

Click Continue to complete this lab.

Summary

In this lab, you learned how to check if kernel performance counters are active in Linux. You started by using the perf stat command to measure performance counters for a simple command like ls, observing the output of various performance events and their counts. This demonstrated the basic functionality of the perf tool for performance analysis.

The lab then guided you through verifying the performance counter configuration within the /sys/kernel filesystem and inspecting counter-related logs in the dmesg output. These steps provided methods to confirm the system's support and active status of kernel performance counters, offering alternative ways to assess their availability beyond the perf stat command.