Introduction
A Pod status such as Pending or CrashLoopBackOff is a symptom, not a root cause. Effective troubleshooting moves through evidence in a consistent order: object configuration, conditions, events, per-container state, logs, and live inspection.
You will diagnose two prepared failures, repair them, and then inspect a healthy multi-container Pod. The goal is a reusable troubleshooting method rather than memorizing one fix.
Use the same order for each incident: start broad, isolate one object, read its current state and events, inspect container-specific evidence, then repair the owning controller. This prevents random changes and helps you explain why a fix worked.
Start with Pod State
In this step, you will identify which workloads are healthy and which need investigation. Begin with a namespace-wide view. -o wide keeps the familiar table but adds useful columns such as Pod IP and assigned node.
cd /home/labex/project/pod-troubleshooting
kubectl get deployments,pods -n practice-debug -o wide
The healthy Pod should show 2/2 Running. One Deployment Pod should show an image pull problem, while the worker should repeatedly restart.
A typical result from the test environment looks like this (Pod suffixes, ages, and restart counts will differ):
NAME READY STATUS RESTARTS AGE
pod/broken-image-58758b4d69-c7j7g 0/1 ImagePullBackOff 0 33s
pod/crashy-worker-5557cfb974-xwpst 0/1 Error 2 33s
pod/web-with-sidecar 2/2 Running 0 33s
Do not worry if the worker currently says Error instead of CrashLoopBackOff. It briefly runs, exits, and waits before Kubernetes starts it again, so the displayed state changes as you watch it. The increasing RESTARTS value is the stronger clue that this container is repeatedly failing.
Display container-level waiting reasons without reading the entire Pod YAML:
kubectl get pods -n practice-debug -o custom-columns='POD:.metadata.name,PHASE:.status.phase,READY:.status.containerStatuses[*].ready,WAITING_REASON:.status.containerStatuses[*].state.waiting.reason,RESTARTS:.status.containerStatuses[*].restartCount'
[*] means “each list item”, so the same table also works for multi-container Pods. Use this first view to decide which diagnostic command comes next.
For example, WAITING_REASON=ImagePullBackOff points toward image configuration or registry access. The worker can show no waiting reason while its latest attempt is Terminated; that is why one table is a starting point, not the complete diagnosis.
At this point, do not repair anything. The image failure and restarting worker are different symptoms and require different evidence. A good first command reduces the search space without assuming the cause.
Diagnose an Image Pull Failure
In this step, you will trace the image failure from status to events. Store the generated Pod name so later commands target the same object:
IMAGE_POD=$(kubectl get pods -n practice-debug -l app=broken-image -o jsonpath='{.items[0].metadata.name}')
echo "$IMAGE_POD"
kubectl describe pod "$IMAGE_POD" -n practice-debug
Read from the bottom upward. The Events section should show that Kubernetes cannot pull nginx:not-a-real-tag. Confirm the configured image directly:
kubectl get pod "$IMAGE_POD" -n practice-debug -o jsonpath='{.spec.containers[0].image}{"\n"}'
kubectl events -n practice-debug --for "pod/${IMAGE_POD}"
Events explain the failed action, while the JSONPath query proves which desired image caused it. Using both avoids mistaking a temporary registry problem for a permanently invalid tag.
In the tested environment, the decisive lines were similar to these:
Image: nginx:not-a-real-tag
State: Waiting
Reason: ErrImagePull
Failed to pull image "nginx:not-a-real-tag": ... manifest unknown
Error: ImagePullBackOff
ErrImagePull records a failed pull attempt. ImagePullBackOff means Kubernetes is now delaying before retrying that same operation. The manifest unknown message and the invalid tag connect the symptom to its cause.
Repair the Deployment rather than editing the generated Pod. kubectl set image changes the Pod template, so the Deployment controller replaces the failed Pod:
kubectl set image deployment/broken-image web=nginx:1.27-alpine -n practice-debug
kubectl rollout status deployment/broken-image -n practice-debug --timeout=60s
The successful end of the command should read:
deployment.apps/broken-image image updated
deployment "broken-image" successfully rolled out
The old Pod is disposable; the Deployment template is the durable desired state. After the rollout, a new Pod name should appear and the Deployment should report one available replica.
Read Current and Previous Logs
In this step, you will diagnose a container that starts and then exits. Find its Pod and inspect restart information:
WORKER_POD=$(kubectl get pods -n practice-debug -l app=crashy-worker -o jsonpath='{.items[0].metadata.name}')
kubectl get pod "$WORKER_POD" -n practice-debug
kubectl describe pod "$WORKER_POD" -n practice-debug
For a restarting container, ordinary kubectl logs reads the current attempt. --previous asks the container runtime for the terminated attempt immediately before it:
kubectl logs "$WORKER_POD" -n practice-debug -c worker
kubectl logs "$WORKER_POD" -n practice-debug -c worker --previous || \
echo "The previous container log is no longer available; continue with the current log and describe output."
The current or previous output should reveal the application's own message:
configuration missing
--previous is especially useful when a new attempt starts before you can inspect the failed one. However, it is historical evidence rather than guaranteed storage: the local runtime may already have removed that container record. If you see unable to retrieve container logs, the fallback message is expected. Combine the available current log with describe, whose Last State, Exit Code, and Restart Count still prove that the process exited repeatedly.
The -c worker option names the container explicitly. Repair the controller by replacing the failing command with a long-running loop:
kubectl patch deployment crashy-worker -n practice-debug --type='json' -p='[{"op":"replace","path":"/spec/template/spec/containers/0/command","value":["sh","-c","echo worker-ready; while true; do sleep 30; done"]}]'
kubectl rollout status deployment/crashy-worker -n practice-debug --timeout=60s
The long patch may look intimidating, so read it from the inside out: path selects the first container's command, op: replace says what kind of change to make, and value supplies the replacement command array. You are changing only one field; the rest of the Deployment remains intact.
The JSON patch precisely replaces one field in the Deployment Pod template. In production, make the equivalent change in the source manifest as well.
Wait for rollout status because the command changes desired state immediately, but replacement and readiness happen asynchronously. A successful API response is not yet proof of a healthy workload.
Target Containers in a Multi-Container Pod
In this step, you will inspect a Pod whose containers have different responsibilities. List their names first:
kubectl get pod web-with-sidecar -n practice-debug -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}'
The JSONPath range iterates through the container list. Read logs from each container independently:
kubectl logs web-with-sidecar -n practice-debug -c reporter --tail=5
kubectl logs web-with-sidecar -n practice-debug -c web --tail=5
--tail=5 limits output to the last five lines. The reporter normally prints repeated reporter-alive messages. NGINX may show startup notices or no request logs yet; either result is normal because nobody has sent it an HTTP request.
Run a command inside the web container:
kubectl exec web-with-sidecar -n practice-debug -c web -- nginx -v
The tested image returned:
nginx version: nginx/1.27.5
The exact patch version can change if the image is refreshed; the important evidence is that the command ran successfully inside the web container.
The first -- ends kubectl options. Everything after it is the command executed in the selected container.
This distinction matters because -c web is an option interpreted by kubectl, while nginx -v runs inside the container. If you omit -c on a multi-container Pod, kubectl may choose a default and show evidence from the wrong process.
Containers in one Pod share its network namespace, so the BusyBox reporter can also request NGINX through the loopback address. In wget -qO-, -q hides progress messages and -O- writes the response body to standard output. The pipe sends that body to head, which keeps only its first lines:
kubectl exec web-with-sidecar -n practice-debug -c reporter -- wget -qO- http://127.0.0.1 | head
The response should begin with <!DOCTYPE html>. This is the reusable diagnostic pattern for later challenges: execute a small HTTP client inside a prepared Pod and read application behavior without exposing a new external port.
Confirm Recovery from Several Angles
In this step, you will verify that fixing the controllers produced healthy replacement Pods. Check controller availability and Pod readiness together:
kubectl get deployments,pods -n practice-debug
kubectl get pods -n practice-debug -o custom-columns='POD:.metadata.name,READY:.status.containerStatuses[*].ready,RESTARTS:.status.containerStatuses[*].restartCount,OWNER:.metadata.ownerReferences[0].name'
Finally, check whether recent warning events still describe active failures:
--field-selector type=Warning asks the API server to return only Warning events. --sort-by=.metadata.creationTimestamp then orders those results from older to newer, so you can distinguish historical failures from the current state.
kubectl get events -n practice-debug --field-selector type=Warning --sort-by=.metadata.creationTimestamp
Old warnings can remain after recovery. Treat the current Pod conditions and controller availability as the source of current health, while events explain how the system reached that state.
After the two repairs, the current-state output should have this shape:
NAME READY UP-TO-DATE AVAILABLE
deployment.apps/broken-image 1/1 1 1
deployment.apps/crashy-worker 1/1 1 1
NAME READY STATUS RESTARTS
pod/broken-image-... 1/1 Running 0
pod/crashy-worker-... 1/1 Running 0
pod/web-with-sidecar 2/2 Running 0
You may still see old Failed, BackOff, or even an early FailedCreatePodSandBox warning from cluster startup. Compare object names and timestamps: the warnings refer to the replaced Pods, while the new Pods are Running and both Deployments are available. Events are a history, not a live health dashboard.
Your final conclusion should combine current evidence: both Deployments are available, all intended containers are ready, and no active Pod remains in an image-pull or crash-loop state. Troubleshooting ends with verification, not with the repair command.
Summary
You diagnosed image and startup failures using conditions, events, container state, current logs, and previous logs. You repaired controller templates instead of generated Pods and learned to target individual containers with logs and exec. The next challenge asks you to apply this evidence-first workflow without a walkthrough.
