How to check if a kernel tracepoint is active in Linux

Introduction

In this lab, we will explore how to check if a kernel tracepoint is active in Linux. Tracepoints are essential for observing kernel events and are invaluable for debugging and performance analysis. We will begin by listing the available tracepoints on your system using the /sys/kernel/debug/tracing filesystem.

Following the initial exploration, we will learn how to check the status of specific tracepoints using the trace-cmd utility and finally verify tracepoints by examining the /proc/kallsyms file.

Skills Graph

List tracepoints with ls /sys/kernel/debug/tracing

In this step, we will begin exploring Linux tracepoints. Tracepoints are hooks placed in the kernel source code that allow you to observe specific events happening within the kernel. They are invaluable for debugging and performance analysis.

The Linux kernel provides a tracing framework, often accessed through the /sys/kernel/debug/tracing directory. This directory contains various files that control and provide information about tracing.

To see the available tracepoints on your system, you can list the contents of the events directory within the tracing filesystem.

Open your terminal if it's not already open. Remember, you can find the Xfce Terminal icon on the left side of your desktop.

Now, type the following command and press Enter:

ls /sys/kernel/debug/tracing/events

This command uses ls to list the contents of the /sys/kernel/debug/tracing/events directory. The events directory is where the kernel exposes the available tracepoints, organized by subsystem.

You will see a list of directories, each representing a kernel subsystem (like syscalls, sched, ext4, etc.). Inside these directories are the actual tracepoint names.

For example, you might see output similar to this (the exact output will vary depending on your kernel version and configuration):

block/      ext4/       kmem/       net/        pci/        random/     sock/       task/       workqueue/
bpf/        fib/        kvm/        nfs/        power/      raw_syscalls/ signal/     timer/      xfs/
compaction/ ftrace/     libata/     nipi/       printk/     regmap/     skb/        udp/        xen/
cpu-freq/   gpio/       mdio/       oom/        qdisc/      rpm/        snd/        vmscan/
cgroup/     header_event  header_page  irq/        migrate/    pagemap/    ras/        scsi/       spi/        writeback/
dma_fence/  i2c/        kcsan/      module/     perf_events/  rcu/        sfc/        syscalls/   xhci-hcd/
enable      iommu/      kprobe/     mpx/        powerpc/    regulator/  signal/     tcp/        xilinx-vsec/
exceptions/ iov/        kscan/      napi/       probe/      rpm/        skb/        timer/      xfs/

This output shows the different categories of events you can trace. In the next steps, we will learn how to check the status of these tracepoints.

Click Continue to proceed.

Check tracepoint status with trace-cmd

In the previous step, we listed the available tracepoints by exploring the /sys/kernel/debug/tracing/events directory. Now, let's use the trace-cmd utility to get more detailed information about tracepoints, specifically their status (whether they are enabled or disabled).

The trace-cmd command is a powerful tool for interacting with the Linux tracing framework. If trace-cmd is not already installed, you can install it using apt.

First, update the package list:

sudo apt update

Then, install trace-cmd:

sudo apt install trace-cmd

You might see output indicating that trace-cmd is already installed, which is fine.

Now, to list all available tracepoints and their status, use the trace-cmd list -e command. The -e option tells trace-cmd to list events (tracepoints).

Type the following command and press Enter:

trace-cmd list -e

This command will output a long list of tracepoints, showing their subsystem and name, followed by their current status in square brackets ([enabled] or [disabled]).

You will see output similar to this (again, the exact list will vary):

  block:block_bio_backmerge [disabled]
  block:block_bio_bounce [disabled]
  block:block_bio_complete [disabled]
  block:block_bio_frontmerge [disabled]
  block:block_bio_queue [disabled]
  block:block_bio_remap [disabled]
  block:block_dirty_buffer [disabled]
  block:block_getrq [disabled]
  block:block_plug [disabled]
  block:block_rq_complete [disabled]
  block:block_rq_insert [disabled]
  block:block_rq_issue [disabled]
  block:block_rq_remap [disabled]
  block:block_rq_requeue [disabled]
  block:block_sync_buffer [disabled]
  block:block_touch_buffer [disabled]
  block:block_unplug [disabled]
  bpf:bpf_trace_printk [disabled]
  bpf:bpf_trace_vprintk [disabled]
  ... (many more tracepoints)

As you can see, most tracepoints are disabled by default to avoid performance overhead. You would typically enable specific tracepoints when you need to trace particular kernel events.

Using trace-cmd list -e is a convenient way to see the full list of tracepoints and their current state without manually navigating the /sys/kernel/debug/tracing filesystem.

Click Continue to move to the next step.

Verify tracepoints in /proc/kallsyms

In the previous steps, we learned how to list tracepoints using the tracing filesystem and the trace-cmd utility. Now, let's explore another way to see tracepoint information: the /proc/kallsyms file.

The /proc/kallsyms file contains the addresses and names of all exported kernel symbols, including functions and variables. Tracepoints are also represented as symbols in this file.

You can use commands like cat and grep to search for tracepoint symbols within /proc/kallsyms. Tracepoint symbols typically follow a naming convention, often including tracepoint or _tracepoint.

Let's try to find symbols related to tracepoints in /proc/kallsyms. We'll use cat to read the file and pipe the output to grep to search for lines containing the word "tracepoint".

Type the following command and press Enter:

cat /proc/kallsyms | grep tracepoint

This command will display lines from /proc/kallsyms that contain the string "tracepoint". You will see output similar to this:

...
ffffffffXXXXXXXX R __tracepoint_module_load
ffffffffXXXXXXXX R __tracepoint_module_free
ffffffffXXXXXXXX R __tracepoint_module_get
ffffffffXXXXXXXX R __tracepoint_module_put
ffffffffXXXXXXXX R __tracepoint_module_request_module
ffffffffXXXXXXXX R __tracepoint_module_module_init
ffffffffXXXXXXXX R __tracepoint_module_module_exit
ffffffffXXXXXXXX R __tracepoint_module_module_request
ffffffffXXXXXXXX R __tracepoint_module_module_autoload
ffffffffXXXXXXXX R __tracepoint_module_module_kset_reg
ffffffffXXXXXXXX R __tracepoint_module_module_kset_unreg
...

The XXXXXXXX represents hexadecimal addresses, which will be different on your system. The important part is the symbol name, which often starts with __tracepoint_ followed by the subsystem and event name.

This confirms that tracepoints are indeed represented as symbols within the kernel's symbol table, accessible through /proc/kallsyms. While /sys/kernel/debug/tracing and trace-cmd provide a more user-friendly interface for managing and viewing tracepoints, understanding that they exist as kernel symbols is fundamental to advanced kernel tracing concepts.

You have now learned three different ways to identify and inspect tracepoints on a Linux system.

Click Continue to complete this lab.

Summary

In this lab, we began by exploring Linux tracepoints, which are crucial for kernel debugging and performance analysis. We learned that the /sys/kernel/debug/tracing directory provides access to the kernel's tracing framework. Specifically, we used the command ls /sys/kernel/debug/tracing/events to list the available tracepoints, which are organized by kernel subsystem within the events directory. This initial step demonstrated how to identify the various kernel events that can be traced on a system.