Multi-Cluster DNS: external-dns and Route 53 Weighted Routing

Running one Kubernetes cluster teaches you about pods. Running the same service across several clusters, in different regions and sometimes different cloud accounts, teaches you about DNS — because that is where the multi-cluster illusion is either maintained or broken. A user types one hostname. Behind it, two or three clusters each own a load balancer with a different address, in a zone that has to be updated by machines, correctly, every time a deployment moves. Get the DNS wrong and "multi-region" becomes "one region plus some unreachable infrastructure."

This is a field guide to the piece that makes it work: external-dns writing records into Amazon Route 53, and weighted routing to spread a single hostname across clusters without collisions.

The shape of the problem

A multi-cluster deployment produces two kinds of hostname. There is the main URL — app.example.com — that users hit and that must resolve across all the clusters serving the app. And there is a per-cluster URL — app-<cluster>.example.com — that each cluster owns individually, useful for targeting one cluster directly and, more importantly, for letting each cluster manage its own record without stepping on its neighbors.

The controller that turns Kubernetes state into DNS records is external-dns. It watches Services and Ingresses, reads their hostnames and the load-balancer address Kubernetes assigned, and reconciles matching records in your DNS provider. Crucially it is upsert-oriented: it creates and updates records to match cluster state, and it uses a companion TXT ownership record to know which records belong to it. That ownership marker — a TXT record carrying heritage=external-dns and an owner ID — is how one cluster's controller avoids clobbering records another controller owns. If you ever debug "my record keeps reverting," the first thing to check is which owner ID is on that TXT.

Why the naive setup collides

The intuitive approach — point every cluster's external-dns at the same app.example.com and let them fight it out — fails in a specific, instructive way. Route 53 will not hold two plain records for the same name and type. If you try to distinguish them with latency-based routing but two clusters sit in the same region, they collide on the latency set identifier and Route 53 rejects the change with InvalidChangeBatch. The record simply never gets written, and that cluster's slice of traffic goes nowhere.

The fix is to make each cluster's contribution to the shared name a distinct routing record. Two approaches work:

• Latency-based routing sends a resolver to the region with the lowest measured latency. It is the right default when your clusters are in genuinely different regions and you want users routed to the nearest one. Each region-cluster gets its own latency record for the shared name.
• Weighted routing assigns each record a weight and splits resolution proportionally. This is what you want when clusters share a region, or when you want explicit control — a canary cluster at weight 10 against a stable cluster at weight 90, for instance. Each cluster's record carries a unique set identifier, so two same-region clusters no longer collide.

The rule of thumb: latency for "nearest wins," weighted for "I decide the split" — and weighted is the escape hatch when two leaves would otherwise share a routing key.

The placeholder-and-adopt pattern

There is a bootstrapping problem hiding here. When a deployment first comes up, its load balancer does not exist yet, so there is no address to put in DNS — but you also do not want the record missing until the LB is ready, because dependent automation (certificate issuance, health checks) wants the name to exist.

The pattern that resolves this: seed a placeholder record at deploy time (a harmless address such as 192.0.2.1, from the reserved documentation range), then let external-dns adopt it once the real load balancer is up, rewriting the record to the live address and keeping it in sync thereafter. The deployer creates the name; the controller makes it true. For this hand-off to work, the placeholder and the controller must agree on ownership — the same TXT owner semantics — or external-dns will treat the placeholder as foreign and refuse to touch it.

Cross-account is a permissions problem, not a DNS problem

A common multi-cluster topology puts clusters in different AWS accounts while the DNS zone lives in one of them. external-dns handles this fine, but only if it can assume a role into the account that owns the zone. Each cluster runs its own external-dns release configured to assume a cross-account role (--aws-assume-role) scoped to that hosted zone. The failure mode when this is missing is silent: the controller runs, sees the Ingress, tries to write the record, gets an access-denied from Route 53, logs it, and moves on — so the deployment looks healthy while its URL never resolves. When a per-cluster URL is dark but the pod is running, suspect IAM before you suspect DNS.

Teardown is where records leak

The unglamorous truth about DNS automation is that creation is easy and deletion is where things rot. Because external-dns is upsert-oriented, a poorly ordered teardown can leave orphan records: the Ingress is deleted, but the controller shuts down before it reconciles the removal, and the record lingers pointing at a load balancer that no longer exists. Multi-cluster makes this worse, because a shared name has several contributing records and only the owning cluster can clean its own.

Two habits prevent it. First, delete the workload and let external-dns observe the removal before tearing down the controller — order matters. Second, when you do a bulk teardown, verify the zone afterward rather than trusting that it drained; a quick check for records still pointing at a deleted LB catches the leaks that automation missed. An alias record that outlives its target is not just clutter — it can serve stale responses or, worse, be adopted by whatever next claims that load-balancer address.

The payoff

When all of this is in place, a single hostname resolves correctly across regions and accounts, moves with your deployments automatically, and cleans up after itself. The user never sees any of it — which is the goal. Multi-cluster DNS is successful precisely when it is invisible, and invisible is a lot of small, correct decisions: distinct routing records per leaf, honest ownership markers, placeholder-then-adopt, cross-account roles that actually have permission, and a teardown order that does not leak. Skip any one and the seams show up as a URL that quietly does not resolve.