Kubernetes troubleshooting
cert-manager: Failed to Finalize Order, 404 Certificate Not Found
A certificate that should take a minute never arrives, and cert-manager reports a 404 from Let's Encrypt even though every challenge passed. The 404 is false and a retry works. The problem is that cert-manager waits an hour before it retries. From cert-manager 1.20 you can cut that to five minutes with one setting.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- The 404 comes from Let's Encrypt, not from your DNS, issuer or account. It is a known, long-standing issue on their side.
- cert-manager treats it as a failed request and waits one hour before trying again, then two, four, and so on up to 32 hours.
- cert-manager 1.20 added --certificate-request-minimum-backoff-duration. Set it to 5m and the first retry runs after five minutes.
- Versions before 1.20 have no setting for this. Trigger a retry by hand with cmctl renew.
The symptom
The Certificate stays at Ready: False. The pod that mounts its secret sits in ContainerCreating, and the deployment times out. The cert-manager controller log shows:
failed to finalize Order resource due to bad request, marking Order as failed
err="404 urn:ietf:params:acme:error:malformed: Certificate not found"And the Certificate itself reports:
The certificate request has failed to complete and will be retried:
Failed to wait for order resource "..." to become ready: order is in
"errored" state: Failed to finalize Order: 404
urn:ietf:params:acme:error:malformed: Certificate not foundThe confusing part is that the domain challenges were already valid. Nothing is wrong with your DNS, your issuer or your account. If the Certificate is stuck earlier than this, with a pending Challenge, see Issuing certificate as Secret does not exist instead.
The cause is on Let's Encrypt's side
Let's Encrypt serves reads from database replicas. When a client creates an object and asks for it again straight away, the second request can reach a replica that has not caught up yet. The answer is a false 404.
A Let's Encrypt community thread from June 2026 describes exactly this error from cert-manager, at about 100 certificates an hour and unrelated to rate limits. The reply there calls it a long-standing known issue and says the client should retry. Let's Encrypt has tracked the underlying problem in its server, Boulder, since 2022: the goal is to turn these 404s into an answer that tells the client a retry may succeed. Both are linked under Sources.
So a retry a few seconds later works. The trouble is when the retry happens.
Why one bad answer costs an hour
cert-manager treats a 4xx answer from the certificate authority as a final failure of that request. After a failed request it waits before making a new one: one hour at first, then double each time, up to 32 hours. The wait is deliberate. It stops a misconfigured cluster from hammering the authority and using up rate limits. For a transient 404 it is far too long, because whatever was waiting for the certificate has usually given up by then.
You can see the state on the Certificate:
kubectl get certificate app-tls -n prod \
-o jsonpath='{.status.lastFailureTime}{" attempts="}{.status.failedIssuanceAttempts}{"\n"}'
kubectl logs -n cert-manager deploy/cert-manager | grep 'Backing off from issuance'The fix: shorten the first retry (cert-manager 1.20 and later)
cert-manager 1.20 added a controller flag for the first retry delay, --certificate-request-minimum-backoff-duration (default 1h). cert-manager 1.21 added a matching flag for the upper limit, --certificate-request-maximum-backoff-duration (default 32h). With the Helm chart:
extraArgs:
- --certificate-request-minimum-backoff-duration=5mRetries then run after 5, 10, 20 and 40 minutes, and so on. It is still a backoff, so a setup that is really broken does not turn into a storm of requests. Versions before 1.20 have no such setting: the one-hour wait is fixed.
We measured it on a test cluster with Pebble, Let's Encrypt's test server, returning a 4xx for an order:
- cert-manager 1.17.4: the next attempt was scheduled 3,600 seconds after the failure.
- cert-manager 1.20.4 with the flag at 5m: the second attempt ran 301 seconds after the first failure, and the third 602 seconds after the second.
- The upgrade from 1.17.4 to 1.20.4 was an in-place helm upgrade that took 12 seconds. The CRDs were kept, and a certificate that was already issued kept its serial number, so it was not issued again.
If you are stuck right now
- Trigger a new attempt instead of waiting: cmctl renew <certificate> -n <namespace>.
- Check what cert-manager thinks: kubectl describe certificate <name>, then read the Issuing condition and lastFailureTime.
- A rate limit error is a different problem. Do not retry in a loop. Fix the cause and wait for the window to pass.
For the limits themselves, see the Let's Encrypt rate limit guide.
Before you upgrade cert-manager
- cert-manager 1.20 supports Kubernetes 1.32 to 1.35, and 1.21 supports 1.33 to 1.36. Check your cluster version first.
- cert-manager 1.18 changed two defaults: private keys are rotated on every renewal, and only one old CertificateRequest is kept.
- cert-manager 1.21 removes some default Helm RBAC. Read its release notes if an issuer uses serviceAccountRef.
- cert-manager recommends upgrading one minor version at a time. Our direct 1.17 to 1.20 upgrade worked in a test cluster and on one production cluster; test yours before you rely on that.
How SeaGit handles this
We hit this on our own cluster: a deployment timed out waiting for a certificate whose challenges had all passed. We traced it to the 404 above, tested the setting, and shipped it in the cert-manager add-on.
- On clusters running Kubernetes 1.32 or later, the SeaGit cert-manager add-on installs cert-manager 1.20.4 with the five-minute first retry already set. Clusters on older Kubernetes stay on 1.17.4, because 1.20 does not support them.
- An existing cluster moves to the new version the next time its cert-manager add-on is applied.
- While a deployment waits for a certificate, SeaGit says so: "is waiting for its TLS certificate", instead of a generic volume error.
If a certificate still does not arrive, contact support with the deployment name.