Troubleshooting Workloads

Introduction

Deployments do not always go as planned. When things break, you need a repeatable troubleshooting method.

In this lab, you will solve "The Case of the Missing Image".

In Kubernetes, troubleshooting often starts with object status, then events, then more detailed inspection. This lab focuses on that evidence-driven workflow.

You will:

• Create a Crime Scene: Intentionally deploy a broken Pod.
• Investigate: Use "binoculars" (get) to see that something is wrong.
• Gather Evidence: Use a "magnifying glass" (describe) to find the root cause.
• Solve the Case: Fix the configuration and watch the application recover.

Create a Broken Pod

We will start by deploying a Pod with a deliberate typo. This simulates a common error: specifying an image that doesn't exist.

This is a good reminder that Kubernetes accepts many manifests whose problems are only discovered later. A YAML file can be syntactically valid while still being operationally wrong. Troubleshooting therefore has to separate "Was the object accepted?" from "Did the workload actually start correctly?"

Create a file named broken-pod.yaml:

nano broken-pod.yaml

Paste the following. Notice the suspicious image tag wrongtag123.

apiVersion: v1
kind: Pod
metadata:
  name: broken-pod
spec:
  containers:
    - name: nginx
      image: nginx:wrongtag123

Save and exit (Ctrl+X, Y, Enter).

Deploy the Pod:

kubectl apply -f broken-pod.yaml

Kubernetes accepts the file because the YAML format is correct. The "crime" hasn't been detected yet.

Observe the Failure

Now, let's look at the status of our deployment. This is like using binoculars to survey the scene.

Statuses such as ErrImagePull and ImagePullBackOff are symptoms, not final explanations. They tell you where to investigate next. Good troubleshooting means treating status output as the first clue rather than the whole diagnosis.

Check the pods in the default namespace:

kubectl get pods

You might first see ContainerCreating, but after a few seconds, it will likely change to ImagePullBackOff or ErrImagePull.

ImagePullBackOff is Kubernetes-speak for: "I tried to pull this image, but I failed. I'm going to back off (wait) for a bit and try again later."

This confirms that the victim (the Pod) is not healthy.

Inspect Events

We know that it failed, but we don't know why. To find out, we need to get close up.

The kubectl describe command is your magnifying glass. It shows you the internal events and logs of a specific resource.

The Events section is especially valuable because it preserves a short narrative from Kubernetes components such as the scheduler and kubelet. When beginners jump straight to random fixes without reading events, they often waste time solving the wrong problem.

Inspect the broken pod:

kubectl describe pod broken-pod

This outputs a lot of data. Scroll down to the very bottom to find the Events section.

You should see something like this:

Type     Reason     Message
----     ------     -------
Normal   Pulling    Pulling image "nginx:wrongtag123"
Warning  Failed     Failed to pull image "nginx:wrongtag123": manifest unknown
Warning  Failed     Error: ErrImagePull

The message manifest unknown is the smoking gun. It means the Docker registry (where images live) has no record of nginx:wrongtag123.

Case solved: The image tag is invalid.

Fix the Pod

Now that we know the root cause, we can fix it.

This final step also illustrates a broader pattern: observe, form a hypothesis from evidence, make one targeted change, and then verify recovery. That disciplined loop is more reliable than making multiple speculative edits at once.

Open the broken-pod.yaml file again:

nano broken-pod.yaml

Correct the "typo". Change nginx:wrongtag123 to just nginx (which defaults to nginx:latest, a valid image).

apiVersion: v1
kind: Pod
metadata:
  name: broken-pod
spec:
  containers:
    - name: nginx
      image: nginx

Save and exit.

Apply the fix. Kubernetes is smart enough to update the existing Pod with the new information.

kubectl apply -f broken-pod.yaml

Now, watch the recovery:

kubectl get pods -w

You should see the status change from ImagePullBackOff to Running.

Press Ctrl+C to stop watching.

Great work, Detective! You identified the error and successfully restored the service.

Summary

In this lab, you learned the fundamental workflow for troubleshooting Kubernetes:

1. Observe: Check kubectl get pods to identify failing workloads.
2. Inspect: Use kubectl describe pod to view the Event log and find error messages.
3. Fix: Update the YAML manifest and apply the correction.

This process—Get, Describe, Fix—will address 90% of the deployment issues you face in your Kubernetes career.