TutorialJuly 16, 20266 views0 comments

Wildcard TLS on Kubernetes with cert-manager and the DNS-01 Challenge

By SeaGit

#cert-manager#tls#lets-encrypt#route53#kubernetes

If you deploy many apps under one domain — app1.example.com, app2.example.com, preview-42.example.com — issuing an individual certificate per hostname stops scaling quickly. You hit Let's Encrypt rate limits, every new subdomain waits on an issuance round-trip, and your ingress controller juggles dozens of secrets. A single wildcard certificate for *.example.com solves all three, and cert-manager automates the whole lifecycle. The catch: wildcards can only be issued through the DNS-01 challenge, and DNS-01 is where the real-world complexity lives.

HTTP-01 vs DNS-01 — why wildcards need DNS-01

Quick refresher on the two challenge types. HTTP-01 proves control of a hostname by serving a token at a well-known URL — easy, but it cannot prove you control every possible subdomain, so no wildcards. DNS-01 proves control of the zone itself by publishing a TXT record at _acme-challenge.example.com. Control of the zone implies control of all its subdomains, so wildcards are allowed. The price is that cert-manager now needs permission to write records into your DNS zone.

Granting Route 53 access with IRSA

On AWS the clean way to grant that permission is IRSA — IAM Roles for Service Accounts. You create an IAM role whose trust policy allows the cert-manager service account (via the cluster's OIDC provider) to assume it, and attach a policy allowing route53:ChangeResourceRecordSets on the hosted zone plus route53:ListHostedZonesByName and route53:GetChange globally. Then the ClusterIssuer's dns01 solver simply names the hosted zone and region; no static keys anywhere. Scope the ChangeResourceRecordSets statement to the single zone ARN — cert-manager does not need account-wide DNS write.

The cross-account case (role chaining)

The setup gets genuinely interesting when the DNS zone lives in a different AWS account than the cluster. This is the normal state of affairs on multi-account platforms: the workload cluster runs in account A, the customer's Route 53 zone lives in account B. The pattern that works is role chaining. The in-cluster IRSA role in account A is allowed to sts:AssumeRole into a small dedicated role in account B, and that foreign role carries the Route 53 permissions. cert-manager supports this directly — the dns01 solver takes a role field, and it will assume that role before touching Route 53.

Two hard-won gotchas from running exactly this in production:

The foreign role must also grant route53:ListHostedZonesByName, not just ChangeResourceRecordSets. cert-manager resolves which zone owns the challenge domain by listing zones by name, and it performs that lookup inside the assumed role. Miss that one permission and you get a misleading error about the zone not being found — the zone is fine, the lookup was forbidden.
Keep the assumable role names deterministic (we template them per cluster and provider) so trust policies can be managed by Terraform instead of by hand, and so a cluster teardown can find and remove exactly the roles it created.

The Certificate resource is the easy part

The Certificate resource itself is anticlimactic after the IAM work: request dnsNames of both example.com and *.example.com in one certificate, point it at the ClusterIssuer, and cert-manager stores the key pair in a Secret your ingress controller references. Renewal is automatic at two-thirds of the ninety-day lifetime. Since every new subdomain is already covered, deploying app number forty adds zero issuance latency — the ingress just references the same wildcard secret.

Operational checks worth wiring up

Operational checks worth wiring up before you need them:

Watch the certificate's expiry as a metric. cert-manager exports certmanager_certificate_expiration_timestamp_seconds, and an alert at twenty days remaining catches every silent renewal failure with time to spare.
When an issuance hangs, describe the Order and Challenge resources cert-manager created; the status conditions name the exact failing step, which beats reading controller logs.
Remember DNS propagation is a real wait. cert-manager polls until the TXT record is visible from public resolvers before asking Let's Encrypt to validate, so slow zone propagation shows up as a pending Challenge, not an error.

One boundary: ingress TLS vs ALB/ACM

One boundary worth stating: this covers certificates your ingress terminates (nginx and friends). If you terminate TLS at an AWS ALB instead, the certificate comes from ACM and cert-manager is not involved — do not point both systems at the same hostname unless you enjoy debugging which one answered.

On SeaGit the wildcard-certificate flow above is automated: enable the certificates add-on and managed DNS, and per-domain issuers, cross-account roles, and renewal are handled for you.

Frequently asked questions

How do I issue a wildcard certificate on Kubernetes?

Use cert-manager with the DNS-01 challenge. Request example.com and *.example.com in one Certificate, give cert-manager permission to write a TXT record to your zone (via IRSA on AWS), and it handles issuance and renewal automatically. HTTP-01 cannot issue wildcards.

Why does cert-manager say my Route 53 zone is not found?

Usually the assumed role is missing route53:ListHostedZonesByName. cert-manager lists zones by name to find the owner of the challenge domain; without that permission the lookup is forbidden and the error misleadingly reports the zone as missing.

Can cert-manager issue certificates when the DNS zone is in another AWS account?

Yes, via role chaining: the in-cluster IRSA role assumes a role in the account that owns the zone. Set the role field on the dns01 solver and ensure the foreign role has both ChangeResourceRecordSets and ListHostedZonesByName.

Do I still need cert-manager if I use an AWS ALB?

No. ALB TLS termination uses certificates from ACM, not cert-manager. Use cert-manager for ingress-terminated TLS (nginx); don't run both for the same hostname.