How to monitor Kubernetes job status and logs

Introduction

Kubernetes is a powerful container orchestration platform that helps manage containerized applications across multiple hosts. One important feature of Kubernetes is its job management system, which allows you to run batch-oriented workloads to completion. In this tutorial, you will learn how to monitor the status of Kubernetes jobs and analyze their logs, essential skills for ensuring your containerized applications run smoothly.

By the end of this tutorial, you will be able to create Kubernetes jobs, check their status, and examine their logs to troubleshoot issues effectively.

Setting Up Your Kubernetes Environment

Before we can monitor Kubernetes jobs, we need to set up a working Kubernetes environment. We'll use Minikube, a tool that creates a single-node Kubernetes cluster on your local machine.

Installing Minikube

Minikube is already installed on your VM. Let's verify the installation by checking the version:

minikube version

You should see output similar to this:

minikube version: v1.29.0
commit: ddac20b4b34a9c8c857fc602203b6ba2679794d3

Starting Minikube

Now let's start Minikube to create a local Kubernetes cluster:

minikube start --driver=docker

This command will take a few minutes to complete. You'll see a series of messages as Minikube downloads necessary components and starts the cluster.

Once complete, you should see a message like:

🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Verifying Kubernetes Installation

Let's verify that our Kubernetes cluster is running correctly by checking the node status:

kubectl get nodes

You should see output similar to this:

NAME       STATUS   ROLES           AGE     VERSION
minikube   Ready    control-plane   1m      v1.26.3

Also, check that the core Kubernetes components are running:

kubectl get pods -n kube-system

You should see several pods running, including components like kube-apiserver, kube-controller-manager, and others.

Now that we have a running Kubernetes cluster, we're ready to create and monitor jobs.

Creating Your First Kubernetes Job

In this step, we'll create a simple Kubernetes job that will run a container, perform a task, and then complete. This will give us something to monitor in the following steps.

Understanding Kubernetes Jobs

A Kubernetes Job is a controller that creates one or more pods and ensures they successfully terminate. Jobs are useful for batch processes, one-time tasks, or any task that should run to completion rather than indefinitely.

Creating a Job Configuration File

Let's create a simple job configuration file. Open a terminal and create a new file called hello-job.yaml in your project directory:

cd ~/project
nano hello-job.yaml

Copy and paste the following YAML content into the file:

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  template:
    spec:
      containers:
        - name: hello
          image: busybox:1.28
          command: ["sh", "-c", 'echo "Hello, Kubernetes!" && sleep 5']
      restartPolicy: Never
  backoffLimit: 4

This configuration defines a job named hello-job that:

• Uses the busybox:1.28 container image
• Runs a command that prints "Hello, Kubernetes!" and then sleeps for 5 seconds
• Has a restart policy of Never, meaning it won't restart after completion
• Has a backoffLimit of 4, meaning it will retry up to 4 times if it fails

Save the file by pressing Ctrl+O, then Enter, and exit nano with Ctrl+X.

Creating the Job

Now let's create the job in our Kubernetes cluster:

kubectl apply -f hello-job.yaml

You should see the output:

job.batch/hello-job created

Congratulations! You've just created your first Kubernetes job. In the next step, we'll learn how to monitor its status.

Monitoring Kubernetes Job Status

Now that we have a job running in our Kubernetes cluster, let's learn how to monitor its status. Monitoring job status is crucial for understanding whether your jobs are completing successfully or encountering errors.

Checking Job Status with kubectl

The primary tool for monitoring Kubernetes resources is kubectl. Let's use it to check the status of our job:

kubectl get jobs

You should see output similar to this:

NAME        COMPLETIONS   DURATION   AGE
hello-job   1/1           10s        30s

This output shows:

• NAME: The name of the job
• COMPLETIONS: The number of completed pods / desired completions
• DURATION: How long the job took to run
• AGE: How long since the job was created

If your job shows 0/1 under COMPLETIONS, it might still be running or has failed. Wait a few seconds and run the command again.

Getting Detailed Job Information

To get more detailed information about a job, use the describe command:

kubectl describe job hello-job

This command provides extensive information about the job, including:

• Labels and annotations
• Selector details
• Parallelism and completion requirements
• Pod statuses
• Events related to the job

Look for the Events section at the bottom, which shows important events like pod creation and completion.

Understanding Job Status

A job can have several statuses:

• Active: The job is still running
• Completed: The job has completed successfully
• Failed: The job has failed after reaching its backoff limit

Let's check if our job's pod has completed:

kubectl get pods

You should see something like:

NAME              READY   STATUS      RESTARTS   AGE
hello-job-abcd1   0/1     Completed   0          1m

The STATUS column shows Completed, indicating that our job ran successfully.

Creating a Job That Takes Longer

Let's create another job that will take longer to complete so we can observe it in the Active state:

cd ~/project
nano long-job.yaml

Copy and paste the following YAML content:

apiVersion: batch/v1
kind: Job
metadata:
  name: long-job
spec:
  template:
    spec:
      containers:
        - name: long
          image: busybox:1.28
          command:
            [
              "sh",
              "-c",
              'echo "Starting long job..." && sleep 30 && echo "Long job completed!"'
            ]
      restartPolicy: Never
  backoffLimit: 4

Save the file and exit nano. Then create the job:

kubectl apply -f long-job.yaml

Now let's check its status immediately:

kubectl get jobs

You should see that long-job shows 0/1 completions because it's still running. If you keep checking every few seconds, you'll eventually see it change to 1/1 after about 30 seconds.

This demonstrates how you can monitor the progress of your jobs in real-time using kubectl.

Analyzing Kubernetes Job Logs

Being able to view and analyze the logs from your Kubernetes jobs is essential for debugging and understanding job behavior. In this step, we'll explore how to access and analyze logs from the jobs we've created.

Getting Pod Names for Our Jobs

Before we can view logs, we need to know the names of the pods created by our jobs. Each job creates one or more pods with names that include the job name and a random suffix.

Let's list all pods related to our jobs:

kubectl get pods --show-labels

This will show you all pods along with their labels. Look for pods with labels like job-name=hello-job or job-name=long-job.

Alternatively, you can filter pods by job name:

kubectl get pods -l job-name=hello-job

This will show only pods that belong to the hello-job job.

Viewing Job Logs

Now that we know our pod names, we can view the logs for our jobs. Use the following command, replacing <pod-name> with the actual name of your pod:

kubectl logs <pod-name>

For example, if your pod is named hello-job-abcd1, you would run:

kubectl logs hello-job-abcd1

You should see the output:

Hello, Kubernetes!

This is the message that our job was programmed to output.

Let's also check the logs of our longer job. First, find the pod name:

kubectl get pods -l job-name=long-job

Then view its logs:

kubectl logs <long-job-pod-name>

You should see:

Starting long job...
Long job completed!

Viewing Logs of Completed Jobs

One of the advantages of Kubernetes jobs is that you can view the logs even after the job has completed. This is very useful for debugging and auditing purposes.

Let's create a job that will fail, so we can see how to debug it:

cd ~/project
nano failed-job.yaml

Copy and paste the following YAML content:

apiVersion: batch/v1
kind: Job
metadata:
  name: failed-job
spec:
  template:
    spec:
      containers:
        - name: failed
          image: busybox:1.28
          command: ["sh", "-c", 'echo "Attempting task..." && exit 1']
      restartPolicy: Never
  backoffLimit: 2

This job will always exit with status code 1, which indicates failure. Save the file and exit nano, then create the job:

kubectl apply -f failed-job.yaml

Wait a few moments for the job to attempt and fail a few times (up to the backoff limit). Then check its status:

kubectl get jobs failed-job

You should see that it shows 0/1 completions and has reached the completion deadline.

Now let's examine what went wrong by checking the logs of the failed pods:

kubectl get pods -l job-name=failed-job

You'll see several pods, all in the Error state. Choose one and view its logs:

kubectl logs <failed-pod-name>

You should see:

Attempting task...

The pod logs show that the task started but then exited with an error code. This information is crucial for debugging job failures.

Following Logs in Real-Time

If you want to follow the logs of a running job in real-time, you can use the -f flag:

kubectl logs -f <pod-name>

This is particularly useful for longer-running jobs where you want to see the output as it happens.

Let's create another long-running job to demonstrate:

cd ~/project
nano counter-job.yaml

Copy and paste the following YAML content:

apiVersion: batch/v1
kind: Job
metadata:
  name: counter-job
spec:
  template:
    spec:
      containers:
        - name: counter
          image: busybox:1.28
          command:
            [
              "sh",
              "-c",
              'for i in $(seq 1 5); do echo "Count: $i"; sleep 5; done'
            ]
      restartPolicy: Never

Save the file and exit nano, then create the job:

kubectl apply -f counter-job.yaml

Now let's follow its logs. First, find the pod name:

kubectl get pods -l job-name=counter-job

Then follow its logs:

kubectl logs -f <counter-job-pod-name>

You'll see the count incrementing every 5 seconds:

Count: 1
Count: 2
Count: 3
Count: 4
Count: 5

Press Ctrl+C to stop following the logs.

By understanding how to access and analyze logs, you can effectively troubleshoot and debug your Kubernetes jobs.

Advanced Job Monitoring Techniques

Now that you know the basics of monitoring Kubernetes jobs, let's explore some more advanced techniques that can help you monitor jobs more effectively, especially in production environments.

Using Labels for Job Organization

Labels are key-value pairs that can be attached to Kubernetes resources. They are useful for organizing and selecting subsets of resources. Let's create a job with custom labels:

cd ~/project
nano labeled-job.yaml

Copy and paste the following YAML content:

apiVersion: batch/v1
kind: Job
metadata:
  name: labeled-job
  labels:
    department: engineering
    app: demo
    environment: training
spec:
  template:
    spec:
      containers:
        - name: labeled
          image: busybox:1.28
          command: ["sh", "-c", 'echo "This is a labeled job" && sleep 10']
      restartPolicy: Never

Save the file and exit nano, then create the job:

kubectl apply -f labeled-job.yaml

Now you can filter jobs by their labels:

kubectl get jobs -l department=engineering

This will show only jobs with the label department=engineering.

You can use multiple labels for more specific filtering:

kubectl get jobs -l department=engineering,environment=training

Monitoring Job Events

Kubernetes generates events for various state changes in your jobs. These events can provide valuable information about the lifecycle of your jobs.

Let's view the events related to our jobs:

kubectl get events --sort-by=.metadata.creationTimestamp

This command shows all events in your namespace, sorted by time. You can filter for events related to a specific job:

kubectl get events --field-selector involvedObject.name=labeled-job

These events can help you understand when and why jobs were created, started, or encountered issues.

Using JSONPath for Custom Output

Kubernetes allows you to use JSONPath to extract specific fields from the output of kubectl commands. This is useful for focusing on particular aspects of your jobs:

kubectl get job labeled-job -o jsonpath='{.status.succeeded}'

This command will output the number of succeeded pods for the labeled-job.

To get the creation time of a job:

kubectl get job labeled-job -o jsonpath='{.metadata.creationTimestamp}'

Creating a CronJob for Scheduled Execution

For tasks that need to run on a schedule, Kubernetes provides CronJobs. Let's create a simple CronJob:

cd ~/project
nano simple-cronjob.yaml

Copy and paste the following YAML content:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: simple-cronjob
spec:
  schedule: "*/1 * * * *" ## Run every minute
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: hello
              image: busybox:1.28
              command: ["sh", "-c", 'date; echo "Hello from CronJob"']
          restartPolicy: Never

Save the file and exit nano, then create the CronJob:

kubectl apply -f simple-cronjob.yaml

Check that the CronJob was created:

kubectl get cronjobs

You should see output similar to:

NAME             SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
simple-cronjob   */1 * * * *   False     0        <none>          10s

Wait a minute or two, then check for jobs created by the CronJob:

kubectl get jobs

You should see jobs with names like simple-cronjob-<timestamp>.

To view the logs of the most recent CronJob execution, first find the pod:

kubectl get pods --sort-by=.metadata.creationTimestamp

Then view its logs:

kubectl logs <latest-cronjob-pod>

CronJobs are powerful for scheduling recurring tasks in your Kubernetes cluster, and the same monitoring techniques we've learned can be applied to them as well.

Cleaning Up

Before we finish, let's clean up the resources we created:

kubectl delete job hello-job long-job failed-job counter-job labeled-job
kubectl delete cronjob simple-cronjob

This will remove all the jobs and the CronJob from your cluster.

Summary

In this tutorial, you have learned how to effectively monitor Kubernetes jobs and analyze their logs. You now know how to:

• Set up a Kubernetes environment using Minikube
• Create Kubernetes jobs with various configurations
• Monitor job status using kubectl get and kubectl describe
• Access and analyze job logs using kubectl logs
• Use advanced techniques like labels, events, and JSONPath for more effective monitoring
• Create scheduled jobs using CronJobs

These skills are essential for managing containerized applications in a Kubernetes environment, allowing you to ensure the reliability and performance of your batch processes. By understanding how to track job progress and troubleshoot issues through logs, you can maintain efficient and error-free Kubernetes workloads.

As you continue your Kubernetes journey, consider exploring more advanced topics like resource management, job parallelism, and integrating with external monitoring solutions like Prometheus and Grafana.