Introduction
The previous labs introduced logs, events, kubectl exec, port-forwarding, and probes. A CrashLooping application combines several of those skills. The Pod starts, the container exits, Kubernetes restarts it, and you need to read the evidence before changing anything.
In this lab, you will deploy an application that exits because a required environment variable is missing. You will observe the restart behavior, read current and previous logs, inspect events, identify the missing configuration in the Deployment, then fix the Deployment and verify that the application becomes reachable.
Prepare a CrashLoop Debugging Namespace
In this step, you will prepare a clean namespace for the CrashLoop debugging exercise.
Start in /home/labex/project. You will create and update the application manifest in this directory.
mkdir -p /home/labex/project
cd /home/labex/project
Check that the Kubernetes API server is reachable. If the VM has just started and the node is not ready yet, wait a few seconds and run the command again.
kubectl get nodes --request-timeout=5s
Expected output includes:
NAME STATUS ROLES AGE VERSION
labex-v135 Ready control-plane ... v1.35...
Wait until the cluster networking Pod is Ready. The fixed application will later be tested through a Service.
kubectl wait --namespace kube-system --for=condition=Ready pod -l k8s-app=calico-node --timeout=120s
Expected output:
pod/calico-node-... condition met
Create a namespace named ckad-prep-crashloop and set it as the current namespace:
kubectl create namespace ckad-prep-crashloop
kubectl config set-context --current --namespace=ckad-prep-crashloop
Confirm the current namespace:
kubectl config view --minify --output 'jsonpath={..namespace}'; echo
Expected output:
ckad-prep-crashloop
Deploy and Observe a CrashLoop
In this step, you will deploy an application that repeatedly exits.
The container script requires the environment variable APP_MESSAGE. If the variable is missing, the script prints a clear error and exits with status 1. Kubernetes will restart the container, which creates the CrashLoop behavior you need to debug.
Create crash-demo.yaml:
cat > crash-demo.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: crash-demo
spec:
replicas: 1
selector:
matchLabels:
app: crash-demo
template:
metadata:
labels:
app: crash-demo
spec:
terminationGracePeriodSeconds: 0
containers:
- name: web
image: busybox:1.36
command:
- /bin/sh
- -c
- |
if [ -z "${APP_MESSAGE:-}" ]; then
echo "startup failed: APP_MESSAGE is required"
exit 1
fi
mkdir -p /www
echo "$APP_MESSAGE" > /www/index.html
httpd -f -p 8080 -h /www
ports:
- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: crash-demo
spec:
selector:
app: crash-demo
ports:
- port: 80
targetPort: 8080
EOF
Apply the manifest:
kubectl apply -f crash-demo.yaml
Expected output:
deployment.apps/crash-demo created
service/crash-demo created
Give Kubernetes a short moment to restart the failing container, then inspect the Pod:
sleep 12
kubectl get pods -l app=crash-demo
Expected output includes a Pod that is not healthy. The exact status can be Error or CrashLoopBackOff depending on timing:
NAME READY STATUS RESTARTS AGE
crash-demo-... 0/1 CrashLoopBackOff ... ...
Store the Pod name and inspect its container state. JSONPath lets you read the specific fields that matter without scrolling through the full object. The first value is the restart count. The second value is the waiting reason, which may still be empty if you catch the Pod while the latest failed container is in an Error state instead of waiting in CrashLoopBackOff.
CRASH_POD="$(kubectl get pods -l app=crash-demo -o jsonpath='{.items[0].metadata.name}')"
echo "$CRASH_POD"
kubectl get pod "$CRASH_POD" -o jsonpath='{.status.containerStatuses[0].restartCount}{" "}{.status.containerStatuses[0].state.waiting.reason}{"\n"}'
Expected output includes a restart count greater than 0. Depending on timing, the reason may be blank or may show a value such as CrashLoopBackOff:
2
or:
... CrashLoopBackOff
Read Logs and Events to Find the Cause
In this step, you will use logs, previous logs, and events to identify why the container exits.
When a container is restarting quickly, kubectl logs may show the current container attempt, while kubectl logs --previous reads the previous terminated container attempt. For CrashLoop debugging, --previous is often the command that exposes the useful startup error.
Read the current logs:
kubectl logs "$CRASH_POD" --container web
Expected output includes:
startup failed: APP_MESSAGE is required
Read the previous container logs. If Kubernetes has already kept the last terminated container log, this shows the same startup error from an earlier attempt. In some fast CrashLoop cases, the runtime may instead report that it cannot retrieve the previous log. That still teaches an important debugging point: use --previous when it is available, but always compare it with the current logs and Pod events.
kubectl logs "$CRASH_POD" --container web --previous
Expected output often includes:
startup failed: APP_MESSAGE is required
It is also acceptable to see a message similar to:
unable to retrieve container logs for ...
Inspect events for the Pod. Events show Kubernetes restarting the failed container.
kubectl describe pod "$CRASH_POD" | sed -n '/Events:/,$p'
Expected output includes events such as BackOff or Started:
Warning BackOff ... Back-off restarting failed container
Now inspect the Deployment's environment configuration. The env list is missing, which explains the log message.
kubectl get deployment crash-demo -o jsonpath='{.spec.template.spec.containers[0].env}'; echo
Expected output is empty:
Fix the Deployment and Verify Recovery
In this step, you will fix the missing environment variable and verify that the application recovers.
The error message named the missing configuration: APP_MESSAGE. Instead of changing the container command, set the required environment variable on the Deployment. Kubernetes will create a new ReplicaSet because the Pod template changes.
Set the environment variable:
kubectl set env deployment/crash-demo APP_MESSAGE=crashloop-fixed
Expected output:
deployment.apps/crash-demo env updated
Wait for the rollout:
kubectl rollout status deployment/crash-demo --timeout=120s
Expected output:
deployment "crash-demo" successfully rolled out
Inspect the new Pod. It should be Running and Ready:
kubectl get pods -l app=crash-demo
Expected output includes:
NAME READY STATUS RESTARTS AGE
crash-demo-... 1/1 Running 0 ...
Create a client Pod and test the Service:
kubectl run crash-client --image=busybox:1.36 --restart=Never --command -- sleep 3600
kubectl wait --for=condition=Ready pod/crash-client --timeout=120s
kubectl exec crash-client -- wget -qO- http://crash-demo
Expected output:
crashloop-fixed
Finally, confirm that the Deployment now includes the environment variable:
kubectl get deployment crash-demo -o jsonpath='{.spec.template.spec.containers[0].env[0].name}{"="}{.spec.template.spec.containers[0].env[0].value}{"\n"}'
Expected output:
APP_MESSAGE=crashloop-fixed
Summary
You debugged a CrashLooping application using the same evidence sources introduced earlier in Stage 5. You observed the failing Pod status, read current and previous container logs, inspected events, identified a missing environment variable in the Deployment, fixed the Pod template with kubectl set env, and verified the recovered application through its Service.
