Kubernetes troubleshooting
Namespace Stuck in Terminating: How to Fix It
A namespace in Terminating is waiting for every object inside it to be removed. Something is still there, either a resource with a finalizer that nothing will clear, or an API service that cannot answer, so the namespace controller cannot finish its sweep. Find the blocker first. Forcing the namespace away can leave orphaned objects behind.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- A namespace is removed only after all namespaced objects in it are deleted. Each stuck object has a finalizer or a controller that is not running.
- An unavailable aggregated API, such as a metrics or custom API service that is down, can stop the sweep even when the namespace looks empty.
- List what remains in the namespace before you change anything. The list tells you which controller to fix.
- Finalizing the namespace by editing the API is the last resort. It leaves any resources it skipped behind.
See what the namespace is waiting for
Read the namespace status. The conditions show whether the sweep failed on discovery, on content or on a finalizer.
kubectl get namespace old-team -o jsonpath='{.status.phase}{"\n"}'
kubectl get namespace old-team -o jsonpath='{range .status.conditions[*]}{.type}{": "}{.status}{" "}{.message}{"\n"}{end}'The condition text is what matters. Match it to one of the causes below. Condition types vary with the Kubernetes version, so copy the exact text from your cluster.
Cause 1: objects with finalizers are still in the namespace
Any namespaced object can hold a finalizer. The namespace cannot finish until each one is removed. List the namespaced types that have objects in the namespace, then look at the finalizers on each.
# Every namespaced resource type that has objects in the namespace
for r in $(kubectl api-resources --verbs=list --namespaced -o name); do
kubectl get "$r" -n old-team --ignore-not-found --no-headers 2>/dev/null \
| sed "s/^/$r: /"
done
# Finalizers on one stuck object
kubectl get <kind> <name> -n old-team -o jsonpath='{.metadata.finalizers}{"\n"}'The cleanest fix is to restore the controller for that finalizer, let it finish, and the namespace will go on its own. If a controller was removed for good, follow the finalizer steps in the pod guide and check each resource it managed.
Cause 2: an aggregated API service is unavailable
Aggregated APIs, such as a metrics API or a custom resource API served by an extension, are registered as APIService objects. When one of them reports Available as False, the namespace controller cannot list its resources, so the sweep cannot be confirmed as complete. The namespace stays in Terminating even if you can see no objects in the core types.
# APIs that are not available, and why
kubectl get apiservices | grep -v True
kubectl describe apiservice <name> | sed -n '/Conditions:/,$p'Restore the service, or remove an API you no longer run
If the extension is still installed, fix its pods or its service so the APIService becomes Available. If the extension was removed and its APIService was left behind, delete the APIService. The namespace then completes the sweep.
kubectl delete apiservice <name>Last resort: finalize the namespace
If the blocker cannot be fixed, you can remove the namespace's kubernetes finalizer through the finalize subresource. This makes the namespace go away even though some objects may remain elsewhere. Do this only after you have confirmed nothing in the namespace still matters, and check for orphaned resources afterwards.
kubectl get namespace old-team -o json > old-team.json
# Edit old-team.json: set "spec": { "finalizers": [] }, then send it to the finalize endpoint
kubectl replace --raw "/api/v1/namespaces/old-team/finalize" -f old-team.json
kubectl get namespace old-team- Delete the old-team.json file afterwards. It is a full copy of the namespace object.
- Objects that were not removed stay in the cluster without a namespace of their own, so list them and clean them up.
- Do not finalize a namespace that holds data volumes you still need, because the volumes may outlive the namespace.
Verify the namespace is gone
Once the blocker is cleared the namespace should disappear. Confirm it is gone, then check there is nothing left that still references it.
kubectl get namespace old-team
kubectl get apiservices | grep -v True