Zero-Downtime Kubernetes Deploys: Readiness, preStop, and Graceful Shutdown

"Zero-downtime deploy" is one of those phrases Kubernetes puts on the box and then quietly makes your problem. The scheduler will happily replace your pods without dropping a Deployment, but whether a request in flight survives that replacement depends on a chain of details that default to "almost right." Almost right means a handful of 502s on every deploy — invisible on a dashboard, extremely visible to the one user who was mid-checkout.

Getting to actually zero requires understanding three moments: when a new pod starts taking traffic, when an old pod stops taking traffic, and what happens to the requests already inside the old pod. Kubernetes gives you a control for each, and the defaults get one of the three wrong.

Readiness is not liveness, and the difference is the whole game

The single most common cause of deploy-time errors is conflating the two probes. A liveness probe answers "should Kubernetes restart this container?" A readiness probe answers "should this pod receive traffic right now?" They look similar and do opposite jobs.

During a rolling update, the endpoint controller adds a pod to a Service's endpoints only once its readiness probe passes. If you have no readiness probe, Kubernetes assumes ready-the-instant-the-process-starts — so traffic arrives before your app has connected to its database, warmed a cache, or loaded config, and those first requests fail. The fix is a readiness probe that checks real dependencies, not just that the port is open:

• Point it at an endpoint that verifies the things a request actually needs (DB reachable, migrations applied, downstream clients constructed).
• Keep periodSeconds tight (a few seconds) so a pod is added promptly once it is genuinely ready.
• Use a separate, cheaper liveness probe so a slow dependency check never triggers a restart loop.

Get this right and the "new pod starts taking traffic" moment becomes safe: the pod is in the load balancer only when it can serve.

The termination race nobody expects

The harder moment is the old pod leaving. When Kubernetes decides to remove a pod it does two things concurrently: it sends the container a SIGTERM, and it tells the endpoint controller to remove the pod from the Service. Those are not ordered. Because endpoint removal propagates through kube-proxy and any external load balancer asynchronously, there is a window where the pod has received SIGTERM but is still listed as a live endpoint — so new connections keep arriving at a process that has started shutting down. That is the classic deploy 502.

The counterintuitive fix is a preStop hook that simply sleeps:

• On preStop, sleep for a small fixed interval (commonly 5–15 seconds) before the app begins shutting down. During that sleep the pod keeps serving normally, which gives the endpoint removal time to propagate everywhere.
• Only after the sleep does SIGTERM reach your process and real shutdown begin. By then the pod is out of rotation, so no new requests are coming.

It feels wrong to add a deliberate delay to a shutdown path, but it is the accepted pattern precisely because Kubernetes does not order endpoint-removal before SIGTERM. You are manually creating the ordering the platform does not guarantee.

Draining what is already inside

The third moment is the requests already being served when shutdown begins. Handling SIGTERM gracefully means: stop accepting new connections, let in-flight requests finish, close idle keep-alive connections, then exit. Most HTTP frameworks have a graceful-shutdown call that does exactly this — the mistake is not wiring it to SIGTERM, so the process dies instantly on the signal and abandons open requests.

Two settings bound this:

• terminationGracePeriodSeconds is the total budget from SIGTERM to the forced SIGKILL. It must comfortably exceed your preStop sleep plus your longest reasonable request. If your preStop sleeps 10 seconds and a slow request takes 20, a 30-second grace period leaves no margin — raise it.
• Your app's own drain timeout should be shorter than the grace period, so you finish on your terms rather than being killed mid-request.

This matters even more on Spot or preemptible nodes, where a reclaim gives you a short, fixed warning before the node disappears. A drain path tuned for graceful shutdown is the same machinery that lets a Spot reclaim leave without dropping connections — which is why platforms that lean on Spot capacity tend to set a generous stop timeout by default.

Rollout knobs: surge and unavailable

The Deployment strategy itself controls how aggressively pods are swapped. maxSurge sets how many extra pods can exist above the desired count during the update; maxUnavailable sets how many can be missing. For zero-downtime you generally want maxUnavailable: 0 and maxSurge at one or more, so new pods come up and pass readiness before any old pod is removed — capacity never dips below target. The tradeoff is transient extra capacity (and cost) during the rollout, which is almost always worth it for a user-facing service.

For a stateful workload that cannot tolerate two versions running at once, the opposite strategy — Recreate — is honest about the brief downtime rather than pretending. Choosing between them is a real decision, not a default to accept blindly.

A working checklist

Put together, "zero downtime" is less a feature than the sum of these settings agreeing with each other:

• Readiness probe that checks real dependencies, tight period.
• Separate liveness probe that will not restart on a slow dependency.
• preStop sleep long enough for endpoint removal to propagate.
• Application SIGTERM handler that drains in-flight requests.
• terminationGracePeriodSeconds > preStop sleep + longest request.
• maxUnavailable: 0, maxSurge ≥ 1 for user-facing rollouts.

None of these is exotic, and that is the point: zero-downtime deploys are not a product you buy but a set of defaults you correct. The reason a managed platform can promise it is that it sets all six consistently, every time — which is exactly the kind of thing humans forget on the one service that mattered.