Introduction
In a production Kubernetes environment, nodes often need maintenance. The challenge is to do that without causing unnecessary workload disruption.
Imagine the Node is a Hotel Room and the Pods are Guests. If you need to renovate the room (perform maintenance), you can't just barge in while guests are sleeping. You need a polite and safe process.
In this lab, you will learn the standard workflow for node maintenance:
- Cordon (Do Not Disturb): Put a sign on the door saying "No New Guests". The hotel (scheduler) stops sending new people to this room.
- Drain (Evict): Politely ask existing guests to leave and move them to another available room.
- Maintenance: Perform the necessary work (renovation).
- Uncordon (Open): Remove the sign and allow scheduling again.
You will be working with a local Minikube cluster to simulate this process.
Cordon the Node
Before performing any maintenance, we must ensure that the Kubernetes scheduler does not place any new pods on the node we are about to work on. This process is called "cordoning". It's like putting a "Do Not Disturb" or "Out of Service" sign on the hotel room door.
This is a scheduling decision, not a shutdown. A cordoned node can still be healthy and still run its existing Pods. Kubernetes is simply marking it as unavailable for new placements so you can begin maintenance in a controlled way.
First, let's identify the name of our node. Run the following command:
kubectl get nodes
You should see one node listed (e.g., minikube) with a status of Ready.
To mark the node as unschedulable, use the cordon command. We will use a command substitution $() to automatically get the node name:
kubectl cordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
Now, verify the status of the node again:
kubectl get nodes
You should see the status change to Ready,SchedulingDisabled. This confirms that the node is now cordoned. No new pods will be scheduled here, but existing pods are still running comfortably.
Drain the Node
Cordoning only stops new pods. We still have running applications (guests) in the room. To safely remove them, we use the drain command.
This command evicts the pods. Kubernetes then realizes the desired number of replicas (2) is not met and attempts to create new replacements on other available nodes.
The word evict matters here. Draining is not meant to be a violent stop. It asks workloads to leave in a Kubernetes-aware way so controllers can replace them and termination rules can be respected. That is why drain is safer than manually deleting random processes on the node.
Execute the drain command. We need to add a few flags to force the eviction in this environment:
--ignore-daemonsets: Some system pods (DaemonSets) are glued to the node and cannot be moved. We ignore them.--delete-emptydir-data: If a pod uses local temporary storage, draining will delete that data. We acknowledge this.--force: Required to delete pods not managed by a ReplicationController, ReplicaSet, Job, DaemonSet, or StatefulSet (like thestorage-provisionerin Minikube).
kubectl drain $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}') --ignore-daemonsets --delete-emptydir-data --force
You will see output indicating that pods are being evicted.
Note: In this single-node Minikube lab, when we drain the only node, the pods will be evicted but will become Pending. This is because there are "no other rooms in the hotel" to move to. In a real multi-node cluster, they would instantly move to a healthy node.
Verify Status
After draining the node, it is important to verify what happened to our applications.
This verification step teaches an important cluster-administration habit: do not stop after issuing a maintenance command. Always inspect the scheduling result from the workload's point of view. A maintenance action is only successful if the applications land where they are supposed to land.
Check the status of your pods:
kubectl get pods
As mentioned, you will likely see the pods in a Pending state.
To understand why they are pending, let's ask Kubernetes. Describe one of the pods:
kubectl describe pod $(kubectl get pods -o jsonpath='{.items[0].metadata.name}')
Look for the Events section at the bottom. You should see a message similar to "0/1 nodes are available: 1 node(s) are unschedulable."
This confirms that the drain was successful. The scheduler wants to run the pod, but it respects the "Do Not Disturb" sign we placed on the only available node.
Uncordon the Node
Imagine you have now finished your "renovation" (OS upgrade or maintenance). It is time to reopen the room and bring the node back into service.
uncordon simply removes the scheduling block. It does not force Pods onto the node immediately. It tells the scheduler that this node is once again eligible for future placement, and then normal reconciliation takes over.
To do this, we use the uncordon command. This removes the "Do Not Disturb" sign.
kubectl uncordon $(kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
Now, verify the node status:
kubectl get nodes
The status should return to Ready (the SchedulingDisabled tag is gone).
Finally, check your pods again. Since the node is open for business, the Kubernetes scheduler should now place the Pending pods back onto this node.
kubectl get pods
You should see them transitioning from Pending to ContainerCreating and finally Running.
Summary
In this lab, you practiced the safe lifecycle of Kubernetes node maintenance. You learned how to:
- Cordon a node to mark it as unschedulable (Do Not Disturb).
- Drain a node to safely evict running workloads.
- Uncordon a node to return it to service.
These commands are essential for systems administrators to ensure high availability. They allow you to repair infrastructure without causing downtime for your applications (in a multi-node cluster, of course!).
