Kubernetes troubleshooting
Readiness Probe Failed in Kubernetes: How to Fix It
A readiness probe failure means the container is running but Kubernetes will not send it traffic. The pod stays Running with READY 0/1, and it is removed from the Service endpoints. Kubernetes does not restart the container for a failed readiness probe, so the fix is in the probe or the app.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- A failed readiness probe does not restart the container. Restarts come from liveness probes. Confusing the two is the most common mistake.
- While the probe fails, the EndpointSlice controller removes the pod IP from the Services that select it. The Service then has fewer or no ready backends.
- The probe event carries the status code or error, which tells you whether the path, the port, the bind address or the timing is at fault.
- For a slow starter, use a startup probe. Do not make the readiness probe wait for a long initial delay.
Read the probe event
Check the pod READY column, then the events. The failure is logged on the pod each time a probe runs and fails.
kubectl get pods -n prod -l app=my-api
kubectl describe pod my-api-7c9f -n prod | sed -n '/Events:/,$p'
kubectl get endpointslices -n prod -l kubernetes.io/service-name=my-api \
-o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{" ready="}{.conditions.ready}{"\n"}{end}'The probe event is the most useful line. Match it to a cause below. The text is illustrative; copy the exact line from your events.
# HTTP probe got a status outside the success range
Readiness probe failed: HTTP probe failed with statuscode: 503
# Nothing answered on that address and port
Readiness probe failed: Get "http://10.0.3.17:8080/healthz": dial tcp 10.0.3.17:8080: connect: connection refused
# The app did not answer within timeoutSeconds
Readiness probe failed: Get "http://10.0.3.17:8080/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)Cause 1: wrong path, port or scheme
The probe asks for a path on a port. If the path is misspelt, the port is the container port but the app listens elsewhere, or the probe uses HTTP where the app serves HTTPS, the probe fails. Check what the app really serves from inside the container:
kubectl exec -n prod my-api-7c9f -- wget -S -O- http://127.0.0.1:8080/healthz
kubectl get pod my-api-7c9f -n prod \
-o jsonpath='{.spec.containers[0].readinessProbe}{"\n"}'For an HTTP probe, the kubelet treats a status code from 200 up to but not including 400 as success, as described on the probes page. A 401 or 403 from an endpoint that requires login therefore fails the probe. Point the probe at a route that does not need a session.
Cause 2: the app listens on 127.0.0.1
A server bound to 127.0.0.1 inside the container answers requests from inside that container only. The kubelet probes the pod IP, so the connection is refused even though the app is running. Bind to 0.0.0.0 or to the pod address.
# Show what the process listens on inside the container
kubectl exec -n prod my-api-7c9f -- ss -ltnp 2>/dev/null || kubectl exec -n prod my-api-7c9f -- netstat -ltnpCause 3: the app is slow, or the probe is too strict
A pod that needs time to warm caches, run migrations or load a model will fail a readiness probe during that time. If the probe also has a short timeout or a low failure threshold, it gives up before the app is ready. Read the timing fields and set them for the app's real startup time:
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 10
timeoutSeconds: 3 # wait longer than the app needs to answer
failureThreshold: 3 # tolerate a few misses before the pod is removed
startupProbe: # protects slow starters from the readiness timing
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 30 # up to 300 seconds to startKeep the readiness check cheap
A readiness endpoint that calls the database or a third-party API fails for every pod at once when that dependency is slow. Check the local state of the app, and let the dependency problem show in your monitoring instead of emptying every Service.
Verify the fix
After you change the probe or the app, the pod should show READY 1/1 and its IP should appear in the Service endpoints again:
kubectl get pods -n prod -l app=my-api -w
kubectl get endpointslices -n prod -l kubernetes.io/service-name=my-api \
-o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{" ready="}{.conditions.ready}{"\n"}{end}'- Restart count still rising? You have a liveness problem too, so check the liveness probe separately.
- Pod ready but the Service still has no backends? Compare the Service selector labels with the pod labels.
- Ready only after a long pause? Raise the startup probe window, not the readiness delay.