Loki Configuration

Introduction

Welcome to this hands-on lab on Loki configuration. Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate.

In this lab, we will focus on the complete, basic workflow of collecting and viewing logs. We will use three key components:

• Loki: The main server responsible for storing and processing logs.
• Promtail: An agent that discovers local log files and ships them to a Loki instance.
• Grafana: A visualization platform where we will connect to Loki to explore and query our logs.

This lab environment has already started Loki and Grafana containers for you. Your task is to configure and run Promtail to send system logs to Loki, and then visualize them in Grafana.

Pull Promtail Docker Image for Log Collection

In this step, you will download the official Docker image for Promtail. Promtail is the agent responsible for discovering log files and sending their contents to Loki. We will run it as a Docker container to keep our setup clean and isolated.

Execute the following command in your terminal to pull the Promtail image from Docker Hub.

docker pull grafana/promtail

You will see output showing the download progress for each layer of the image. Once completed, the image will be available on your local machine.

Using default tag: latest
latest: Pulling from grafana/promtail
Digest: sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Status: Downloaded newer image for grafana/promtail
docker.io/grafana/promtail

Now that we have the image, we can proceed to create a configuration file for it in the next step.

Create promtail.yml to Scrape System Logs

In this step, you will create a configuration file for Promtail. This YAML file tells Promtail where the Loki server is located and which log files it should monitor. All your work should be done in the ~/project directory.

First, create a new file named promtail.yml using the nano editor.

nano promtail.yml

Now, copy and paste the following configuration into the nano editor.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: varlogs
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

Let's break down this configuration:

• server: Configures Promtail's own web server, which is not used in this lab.
• positions: Specifies a file where Promtail records the last read location in each log file, so it doesn't re-send old logs on restart.
• clients: Defines the address of the Loki instance. We use http://loki:3100 because both Promtail and Loki will run on the same Docker network, and loki is the container name.
• scrape_configs: This is the core section. We define a job named varlogs that scrapes all files ending with .log in the /var/log/ directory.

After pasting the content, save the file and exit nano by pressing Ctrl+X, then Y, and finally Enter.

Run Promtail Container Pointing to Loki

In this step, you will start the Promtail container using the configuration file you just created. The command will mount both your configuration file and the host's log directory into the container.

Execute the following docker run command to start Promtail:

docker run -d --name promtail --network monitoring-net -v $(pwd)/promtail.yml:/etc/promtail/config.yml -v /var/log:/var/log grafana/promtail -config.file=/etc/promtail/config.yml

Here's an explanation of the command's flags:

• -d: Runs the container in detached mode (in the background).
• --name promtail: Assigns a name to the container for easy reference.
• --network monitoring-net: Connects the container to the same network as Loki and Grafana.
• -v $(pwd)/promtail.yml:/etc/promtail/config.yml: Mounts your local promtail.yml into the container at the expected location.
• -v /var/log:/var/log: Mounts the host's /var/log directory into the container so Promtail can read the log files.
• -config.file=/etc/promtail/config.yml: Tells the Promtail process inside the container which configuration file to use.

After running the command, Docker will output the unique ID of the new container. You can verify that the container is running with the docker ps command.

docker ps

You should see promtail, loki, and grafana in the list of running containers.

CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                                       NAMES
...            grafana/promtail        "/usr/bin/promtail -c…"   A few seconds ago   Up a few seconds   0.0.0.0:9080->9080/tcp                      promtail
...            grafana/grafana         "/run.sh"                About a minute ago  Up About a minute  0.0.0.0:8080->3000/tcp                      grafana
...            grafana/loki            "/usr/bin/loki -conf…"   About a minute ago  Up About a minute  0.0.0.0:3100->3100/tcp, 9095/tcp            loki

Add Loki as Data Source in Grafana

In this step, you will configure Grafana to use Loki as a data source. This will allow you to query and visualize the logs that Promtail is sending to Loki.

Please follow these instructions carefully:

1. Due to LabEx VM's reverse proxy settings, switch to Desktop Interface, click the Firefox browser in the top left corner, and enter http://localhost:8080 in the address bar. You should see the Grafana login page.

2. Log in to Grafana. Use the username admin and the password admin. You may be asked to change the password; you can skip this for the lab.

3. On the left-hand menu, click the Connections icon (looks like a plug or connector).

4. In the Connections page, click on Data sources.

5. On the Data sources page, click the Add new data source button.

6. From the list of available data source types, select Loki.

7. In the search box, type Loki and click on the Loki data source option that appears.

8. You will be taken to the Loki data source configuration page. In the URL field under the HTTP section, enter the following address:

   http://loki:3100
   

   We use loki as the hostname because the Grafana and Loki containers are on the same Docker network (monitoring-net), and Docker provides DNS resolution between containers on the same network.

9. Scroll to the bottom and click the Save & test button.

If everything is configured correctly, you will see a green banner with the message "Data source is working". You have now successfully connected Grafana to your Loki instance.

Query Basic Logs in Grafana Explore View

In this final step, you will use Grafana's "Explore" view to run a simple query and see the logs that Promtail is shipping from your environment.

1. In the Grafana UI, navigate to the left-hand sidebar and click the compass icon (Explore).

2. At the top left of the Explore page, you will see a dropdown menu for selecting the data source. If it's not already selected, choose Loki.

3. You will see a "Log browser" input field. This is where you write your LogQL queries.

4. Click on the "Code" button to switch to the code editor.

5. Enter the following query into the input field:

   {job="varlogs"}
   

   This LogQL query selects all log streams that have the label job with the value varlogs. We defined this label in our promtail.yml file.

6. Press Shift+Enter or click the blue Run query button on the right side of the screen.

After running the query, you should see log lines appear in the main panel below the query editor. You will see logs from various files in /var/log. This confirms that Promtail is successfully scraping logs and sending them to Loki, and Grafana is able to query them.

Summary

Congratulations! You have successfully configured a basic logging pipeline using Loki, Promtail, and Grafana.

In this lab, you have learned how to:

• Pull and run Docker containers for log processing tools like Promtail.
• Create a Promtail configuration file to specify a Loki server and define log scraping jobs.
• Use Docker networking to connect services (Promtail, Loki, and Grafana).
• Add Loki as a data source within the Grafana user interface.
• Execute a basic LogQL query in Grafana's Explore view to find and inspect your logs.

This setup is the foundation of a powerful, scalable, and cost-effective log aggregation system. From here, you could explore more advanced LogQL queries, log parsing, and setting up alerts in Grafana based on log content.