Prometheus PromQL: Querying Time-Series Data

Introduction

Welcome to this hands-on lab on PromQL basics. Prometheus is a powerful open-source monitoring and alerting toolkit, and its query language, PromQL, is at the heart of its power. PromQL allows you to select and aggregate time-series data in real time.

In this lab, you will work with a pre-configured environment consisting of a Prometheus server that is actively collecting metrics from a Node Exporter. Your goal is to learn the fundamental syntax of PromQL to query this data, starting with simple metric selections and progressing to more advanced queries using functions and filters.

Understand the Provided Environment

In this step, you will familiarize yourself with the lab environment. The setup process has already started two Docker containers for you. You do not need to perform any setup.

The two containers are:

node-exporter: This container's purpose is to expose a wide variety of hardware- and kernel-related metrics from the host machine it's running on.
prometheus: This is the Prometheus server itself. It is configured to "scrape" (collect) the metrics from the node-exporter and store them in its time-series database.

You can verify that both containers are running by executing the docker ps command in the terminal.

docker ps

You should see an output similar to the following, listing both the prometheus and node-exporter containers with a status of "Up".

CONTAINER ID   IMAGE                       COMMAND                  CREATED          STATUS         PORTS                                       NAMES
486db39ca176   prom/prometheus      "/bin/prometheus --c…"   9 seconds ago    Up 8 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus
3f02110c8fde   prom/node-exporter   "/bin/node_exporter"     11 seconds ago   Up 8 seconds   0.0.0.0:9100->9100/tcp, :::9100->9100/tcp   node-exporter

This confirms that your monitoring stack is ready. In the next step, you will access the Prometheus user interface to start querying these metrics.

Access Expression Browser in Prometheus UI

In this step, you will access the Prometheus web UI and locate the Expression Browser, which is the primary tool for running PromQL queries.

The Prometheus container's web server runs on port 9090, and the lab environment automatically makes this accessible to you.

Click the + button in the LabEx interface and choose Web Service.
Enter 9090 for the port and open the new tab that appears.

This action opens the Prometheus UI in a separate tab within your lab environment. By default, you should land on the Graph page. On this page, you will see a text input field labeled Expression. This is the Expression Browser, where you will enter and execute all your PromQL queries for this lab.

Prometheus Expression Browser

Now that you know where to execute queries, you're ready to write your first one.

Execute Basic Query like node_cpu_seconds_total

In this step, you will execute your first basic PromQL query to select a metric. The simplest query in PromQL is just the name of a metric. This will select all time series that have this metric name.

Navigate to the Expression Browser in the Prometheus UI that you opened in the previous step.

In the Expression input field, type the following metric name:

node_cpu_seconds_total

Click the Execute button.

After executing, look at the results in the Table view below the graph. You will see a list of time series.

node_cpu_seconds_total{cpu="0", instance="node-exporter:9100", job="node", mode="idle"}      3456.78
node_cpu_seconds_total{cpu="0", instance="node-exporter:9100", job="node", mode="iowait"}    12.34
node_cpu_seconds_total{cpu="0", instance="node-exporter:9100", job="node", mode="system"}    56.78
...

Let's break down this result:

node_cpu_seconds_total: This is the metric name. It represents a counter that tracks the total number of seconds the CPU has spent in various modes.
{cpu="0", mode="idle", ...}: These are the labels. Labels are key-value pairs that uniquely identify a time series. Here, you can see different series for each CPU core (cpu="0") and each mode (mode="idle", mode="system", etc.).
3456.78: This is the latest value recorded for that specific time series. Since it's a counter, this value will only increase over time.

Note: If you don't see the latest value, click the Labby dialog box in the right corner of the page and close it temporarily.

You have successfully queried Prometheus for all time series of a specific metric.

Apply rate Function and Label Filters

In this step, you will refine your query using label filters and the rate() function to get more meaningful data. The raw value of a counter is often not useful; we are usually more interested in its rate of change.

First, let's filter the results to see only the CPU time spent in idle mode. You can do this by adding a label selector in curly braces {} to your query.

Enter the following query in the Expression Browser and execute it:

node_cpu_seconds_total{mode="idle"}

This query now only returns the time series where the mode label is exactly idle.

Next, let's calculate the per-second rate of change for this counter. For this, we use the rate() function. The rate() function needs to know over what time period to calculate the rate. We specify this with a range vector selector, like [5m] for the last 5 minutes.

Now, modify your query to wrap it in the rate() function and add a 5-minute range selector. Execute the final query:

rate(node_cpu_seconds_total{mode="idle"}[5m])

Let's break down this advanced query:

node_cpu_seconds_total{mode="idle"}: This is the instant vector selector that filters for our desired metric and label.
[5m]: This is the range selector, which tells Prometheus to get all the data points for the last 5 minutes for the selected time series.
rate(...): This is the function that takes the 5-minute range of data and calculates the per-second average rate at which the counter increased.

The result is no longer a large counter value, but a value typically between 0 and 1 (for a single CPU core). A value of 0.95 would mean that over the last 5 minutes, the CPU was, on average, 95% idle. This is a much more useful piece of information for monitoring.

Summary

Congratulations on completing this lab on PromQL basics! You have successfully queried a live Prometheus instance and transformed raw data into meaningful information.

In this lab, you learned how to:

Navigate the Prometheus UI to the Expression Browser.
Execute basic queries to select all time series for a metric.
Understand the structure of a time series, including its metric name and unique set of labels.
Filter time series using label selectors ({...}).
Calculate the rate of change for a counter using the rate() function and a range vector selector ([5m]).

These are the fundamental building blocks for working with Prometheus. From here, you can explore the many other functions and operators that PromQL offers to build powerful and insightful monitoring dashboards and alerts.

PromQL Basics