Kubernetes troubleshooting
Pod Stuck in Terminating in Kubernetes: How to Fix
A pod in Terminating has been asked to stop, and something has not finished. Either a finalizer on the pod is waiting for a controller that is not running, or the node cannot confirm the containers stopped. Find out which before you force anything, because a forced delete skips the cleanup that finalizers exist to do.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- Terminating means the pod has a deletionTimestamp. The API keeps the object until the kubelet confirms the containers stopped and every finalizer is removed.
- The two common blockers are a finalizer whose controller is gone, and a node that is NotReady or unreachable, so the kubelet cannot report back.
- Check the finalizers and the node first. Force deletion is the last step, not the first.
- A force delete removes the pod from the API immediately, so a new pod with the same name can start while the old containers may still be running on the node.
Confirm the pod is stuck and see why
Read the deletion timestamp, the finalizers and the node the pod is on. The pod spec and status hold the answer.
kubectl get pod my-api-7c9f -n prod -o wide
# deletionTimestamp shows when the delete started, and finalizers lists blockers
kubectl get pod my-api-7c9f -n prod \
-o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}{.spec.nodeName}{"\n"}'
kubectl describe pod my-api-7c9f -n prod | sed -n '/Events:/,$p'Decide the branch from the output. A non-empty finalizers list points to a controller. A node that is NotReady, or a pod that stays on a node you cannot reach, points to the node.
Cause 1: a finalizer is waiting for a controller
A finalizer is a key in metadata.finalizers. The controller that owns it does cleanup work, then removes its key. If that controller is not running, is crashing, or was uninstalled before its resources were removed, the key stays and the pod never leaves. Find the owner by its finalizer name, then check that controller.
# Which finalizer names are set
kubectl get pod my-api-7c9f -n prod -o jsonpath='{.metadata.finalizers}{"\n"}'
# Is the controller for that finalizer running? Replace the namespace and name
kubectl get pods -A | grep -i <controller-name>
kubectl logs -n <controller-namespace> deploy/<controller-name> --tail=50Remove a finalizer only when its controller is gone for good
If the controller was removed and nothing will ever clear the key, remove the finalizer by hand. This skips the cleanup that finalizer performed, such as detaching a volume or an external record, so check that resource yourself afterwards.
kubectl patch pod my-api-7c9f -n prod --type=merge \
-p '{"metadata":{"finalizers":null}}'Cause 2: the node cannot confirm the containers stopped
The API waits for the kubelet on the pod's node to report that the containers have stopped. If the node is NotReady, powered off or cut off from the control plane, that report never arrives. The pod stays Terminating until the node returns, or until something removes it from the API.
kubectl get node <node-name> -o wide
kubectl describe node <node-name> | sed -n '/Conditions:/,/Addresses:/p'
# Pods on that node, to see how many are affected
kubectl get pods -A -o wide --field-selector spec.nodeName=<node-name>On managed clusters, a node that is NotReady usually needs its instance or the kubelet service restarted, or the node group replaced. Do that before you delete pods, so you do not strand other workloads.
Last resort: force delete
A force delete with a zero grace period removes the pod from the API at once, and the API does not wait for the kubelet. The kubernetes.io docs call this potentially disruptive. Use it only when the node is gone for good, and when you are sure the workload can run a second copy without conflict, for example a stateless Deployment with no shared volume lock.
kubectl delete pod my-api-7c9f -n prod --grace-period=0 --force- Never force-delete a StatefulSet pod that owns a volume. The replacement can start while the old pod still writes to the disk.
- Check the node has really been removed or replaced before you rely on the force delete.
- After a force delete, confirm no duplicate process is running on any node you could not reach.
Verify the pod is gone and the workload is healthy
A Deployment will create a replacement pod after the old one is removed. Confirm the replacement is ready, and that no old pod is left behind on a node.
kubectl get pods -n prod -l app=my-api -o wide
kubectl rollout status deployment/my-api -n prod