Introduction
You can now describe desired state with manifests and create Pods and Deployments. The next essential skill is understanding what to do when Kubernetes cannot realize that desired state.
In this lab, you will work with two small Deployments: one healthy and one containing an intentional image-tag typo. You will follow a repeatable path from broad symptoms to specific evidence, repair the manifest rather than patching only the live object, and then inspect the recovered application through logs and commands inside its container.
The goal is not to memorize every possible failure. It is to build a calm troubleshooting habit: observe, narrow, inspect evidence, repair desired state, and verify recovery.
Create a Controlled Failure
Real troubleshooting begins with a symptom. In this step, you will deploy a healthy workload and an intentionally broken workload so you can compare them under identical cluster conditions.
Environment startup: This lab starts a complete Kubernetes cluster for you. Configuring its control plane, node, and networking components usually takes 2–3 minutes. Please wait patiently for the environment to finish loading before running the commands below.
Move to the prepared workspace and list its files. cd changes the current directory; ls lists the names inside it. The commands are on separate lines and run in order:
cd /home/labex/project/debug-lab
ls
You should see healthy-web.yaml and broken-web.yaml. Both define one-replica Deployments, but one contains a subtle configuration error that you will diagnose later.
Apply both manifests. kubectl apply sends desired state to the API server, and each -f names one input file. One command can accept multiple -f options:
kubectl apply -f healthy-web.yaml -f broken-web.yaml
Wait for the known-good Deployment first:
kubectl rollout status deployment/healthy-web --timeout=60s
The message deployment "healthy-web" successfully rolled out establishes a useful baseline: the cluster can schedule Pods and run the cached NGINX image.
Now give the other Deployment a short opportunity to roll out:
kubectl rollout status deployment/broken-web --timeout=15s || true
The timeout is expected. || true tells the shell to continue because this failure is evidence for the exercise, not a reason to stop the lab.
Compare the Deployment summary:
kubectl get deployments
healthy-web should show 1/1 ready, while broken-web shows 0/1. You have now established that the problem is workload-specific rather than a total cluster failure.
Narrow the Problem with Resource Summaries
In this step, you will start broad before diving into details. Kubernetes controllers create a chain of objects, so a Deployment problem often becomes visible first in its ReplicaSet and Pod.
List the related object types together. Commas let one kubectl get request several resource types, while -o wide adds useful columns such as node and IP information:
kubectl get deployments,replicasets,pods -o wide
Read the output from top to bottom:
- A Deployment reports the desired and available replica counts.
- A ReplicaSet carries that desired replica count closer to the Pods.
- A Pod reports container readiness and a short status reason.
Filter the view to only the broken application by using its label:
kubectl get pods -l app=broken-web -o wide
The Pod name contains a generated suffix, so labels are safer than copying a changing name into scripts.
Ask for just the fields that matter at this stage. -o custom-columns='...' builds a table from explicit object fields. Each entry has a heading such as NAME, followed by the JSON field path that supplies its value. The trailing backslash joins the two displayed shell lines into one command:
kubectl get pods -l app=broken-web \
-o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[0].ready,WAITING_REASON:.status.containerStatuses[0].state.waiting.reason,NODE:.spec.nodeName'
The waiting reason may initially be ErrImagePull and later become ImagePullBackOff. Both indicate that the container never started because Kubernetes could not obtain its image. This is more precise than merely saying “the Pod is down.”
Inspect the Pod with describe
In this step, you will use describe to learn why the container is waiting. The summary told you what is wrong; now you will gather the explanation.
Store the generated Pod name in a shell variable. $(...) is command substitution: the shell runs the inner kubectl command and assigns its output to BROKEN_POD. JSONPath selects the first matching Pod's name, and echo prints the stored value:
BROKEN_POD=$(kubectl get pods -l app=broken-web -o jsonpath='{.items[0].metadata.name}')
echo "$BROKEN_POD"
Describe that Pod:
kubectl describe pod "$BROKEN_POD"
describe combines useful fields with recent events. Focus on three areas:
- Containers → Image shows the exact requested image.
- State → Waiting → Reason describes the current container state.
- Events records the kubelet's attempts and error messages.
For this scenario, the event message says the tag 1.27-alpine-missing cannot be found. The cluster is doing what the manifest requested; the desired state itself is wrong.
Confirm the image directly with JSONPath:
kubectl get pod "$BROKEN_POD" -o jsonpath='Image: {.spec.containers[0].image}{"\n"}'
JSONPath is helpful when a large YAML or describe output contains more information than you need. Here it isolates the field that must eventually be repaired.
Read Events as a Timeline
In this step, you will read events as a timeline of Kubernetes activity. Events are short-lived diagnostic records that help explain scheduling, image pulls, container starts, restarts, and many other state transitions.
List recent namespace events in chronological order. --sort-by sorts objects by the given metadata field; the quotes keep the JSON-style field path together as one argument:
kubectl get events --sort-by='.metadata.creationTimestamp'
The last rows are usually the newest. Find entries whose OBJECT column refers to the broken Pod and whose REASON includes values such as Pulling, Failed, or BackOff.
You can reduce noise by filtering events to the generated Pod name. --field-selector filters server-side object fields rather than labels. The comma means both conditions must match, and the backslashes continue one command across readable lines:
BROKEN_POD=$(kubectl get pods -l app=broken-web -o jsonpath='{.items[0].metadata.name}')
kubectl get events \
--field-selector involvedObject.kind=Pod,involvedObject.name="$BROKEN_POD" \
--sort-by='.metadata.creationTimestamp'
Think of get, describe, and events as complementary views:
getquickly locates the unhealthy object.describecombines configuration, state, and related events for one object.eventsprovides a time-ordered view that can reveal repeated attempts.
The repeated BackOff entries do not mean Kubernetes has abandoned the Pod. They mean it is spacing out repeated pull attempts after failures.
Repair Desired State and Verify Recovery
In this step, you will repair desired state and verify recovery. You have enough evidence to act: the manifest requests a nonexistent image tag. Repair the saved manifest first, then apply it so the file and live cluster remain consistent.
Show the image lines in both manifests for comparison. grep searches text, -n prefixes each match with its line number, and both filenames are searched in one command:
grep -n 'image:' healthy-web.yaml broken-web.yaml
The healthy manifest uses nginx:1.27-alpine; the broken manifest adds the nonexistent -missing suffix.
Replace only that suffix. sed performs a text substitution written as s/old/new/; -i edits the named file in place instead of only printing the changed text:
sed -i 's/nginx:1.27-alpine-missing/nginx:1.27-alpine/' broken-web.yaml
Validate the repaired file locally:
kubectl apply --dry-run=client -f broken-web.yaml
Preview the difference between the file and live object:
kubectl diff -f broken-web.yaml || true
kubectl diff exits with code 1 when it finds a difference, so || true keeps the learning sequence moving. In the diff, a line beginning with - contains the old image and a line beginning with + contains the repaired image.
Apply the repair and wait for recovery:
kubectl apply -f broken-web.yaml
kubectl rollout status deployment/broken-web --timeout=60s
Confirm both Deployments are now healthy:
kubectl get deployments
Both should report 1/1 ready. Kubernetes created a new ReplicaSet and Pod from the corrected Pod template; you did not need to manually repair the failed Pod.
Read Application Logs
In this step, you will use application logs as a new evidence source now that the container starts. kubectl logs retrieves the container's standard output and standard error streams.
Select the new healthy Pod managed by broken-web. This repeats command substitution and JSONPath from earlier. --field-selector=status.phase=Running adds a server-side requirement so the selected Pod is running:
WEB_POD=$(kubectl get pods -l app=broken-web \
--field-selector=status.phase=Running \
-o jsonpath='{.items[0].metadata.name}')
echo "$WEB_POD"
NGINX may have no access-log entry yet because nobody has requested a page. Generate one from inside the Pod. In kubectl exec POD -- COMMAND, the -- separates kubectl options from the command executed inside the container. wget -qO- fetches quietly and writes the page to standard output; the pipe | passes that output to head, which shows only the beginning:
kubectl exec "$WEB_POD" -- wget -qO- http://127.0.0.1 | head
The HTML begins with <!DOCTYPE html>, proving that NGINX answered locally on port 80.
Now read the recent logs. --tail=10 limits output to the ten most recent lines so startup noise does not overwhelm the useful request entry:
kubectl logs "$WEB_POD" --tail=10
Look for an HTTP request containing GET / HTTP/1.1 and a 200 response code. Logs are most useful when a container runs but the application behaves incorrectly. They are usually not useful for an image-pull failure because the container never started.
Inspect from Inside the Container
In this step, you will inspect the recovered application from inside its container. kubectl exec runs a command in an already running container and can check its filesystem, processes, environment, DNS view, or local network behavior.
Reuse the running Pod name:
WEB_POD=$(kubectl get pods -l app=broken-web \
--field-selector=status.phase=Running \
-o jsonpath='{.items[0].metadata.name}')
Ask the container for its hostname:
kubectl exec "$WEB_POD" -- hostname
The output matches the Pod name because Kubernetes sets the Pod's hostname by default.
Check the NGINX configuration syntax inside the container:
kubectl exec "$WEB_POD" -- nginx -t
The messages syntax is ok and test is successful show that the application configuration is internally valid.
Finally, perform a compact inside-out health check. >/dev/null discards the downloaded HTML, and && runs echo only if wget succeeds. Therefore the success message appears only after an HTTP response is received:
kubectl exec "$WEB_POD" -- wget -qO- http://127.0.0.1 >/dev/null && echo "NGINX responded inside the Pod"
Use exec thoughtfully. It requires a running container, so it could not diagnose the earlier image-pull failure. Your evidence ladder for this incident was:
get -> describe -> events -> repair manifest -> rollout status -> logs -> exec
Different failures may stop at different rungs, but moving from inexpensive summaries toward deeper inspection keeps troubleshooting focused.
Summary
You practiced a complete beginner debugging loop on Kubernetes v1.35. You compared healthy and unhealthy workloads, narrowed the issue with labels and concise fields, used describe and events to identify an invalid image tag, repaired the declarative source of truth, and verified recovery with rollout status, logs, and commands inside the container.
The central lesson is to choose evidence that matches the workload's current lifecycle stage. When a container has not started, inspect state and events. Once it is running, logs and exec can reveal application-level behavior. The next challenge asks you to use this workflow independently.
