DevOpsJuly 16, 20264 views0 comments

Debugging Kubernetes DNS: A Field Guide to CoreDNS, ndots, and the 5-Second Timeout

By SeaGit

#kubernetes#dns#coredns#networking#troubleshooting

Almost every "the network is broken" incident we have debugged on Kubernetes eventually turned out to be DNS. Not because DNS is badly designed, but because Kubernetes layers three separate resolution systems on top of each other — the pod resolver config, CoreDNS, and whatever sits upstream of CoreDNS — and a failure in any layer looks identical from the application: a timeout. This field guide walks the path one hop at a time so the layer that lies to you becomes obvious.

How a Kubernetes DNS lookup actually works

Start with what actually happens on a lookup. Every pod gets an /etc/resolv.conf pointing at the cluster DNS Service IP (usually 10.96.0.10 on kubeadm clusters, or whatever kube-dns ClusterIP your CNI advertises). That Service load-balances to the CoreDNS pods in kube-system. CoreDNS answers cluster names (anything under cluster.local) from the Kubernetes API and forwards everything else to the upstream resolver it inherited from the node — on EKS, the VPC resolver at the VPC CIDR base plus two.

The ndots:5 trap

The single most misunderstood line in that resolv.conf is options ndots:5. It means: if a name has fewer than five dots, try every search domain first before trying the name as-is. So when your app looks up api.stripe.com (two dots), the resolver first asks for api.stripe.com.default.svc.cluster.local, then api.stripe.com.svc.cluster.local, then api.stripe.com.cluster.local, then the node search domains, and only then the real name. That is four to six wasted round trips per lookup, doubled because A and AAAA are queried separately. Under load this alone can saturate CoreDNS.

Two practical fixes. First, for external names your app calls constantly, use a trailing dot (api.stripe.com.) or set a per-pod dnsConfig with ndots:2. Second, keep an eye on the CoreDNS cache plugin — it is enabled in the default Corefile, but the default TTL only helps if queries actually repeat; the ndots expansion generates unique NXDOMAIN misses that pollute the cache.

The famous 5-second DNS timeout

Now the famous 5-second timeout. If lookups intermittently take exactly 5 seconds, you are not seeing a slow server — you are seeing a dropped UDP packet plus the resolver retry timer. On kernels doing conntrack-based NAT there is a well-documented race where two UDP packets (the parallel A and AAAA queries) hit conntrack insertion at the same time and one gets dropped. The glibc resolver waits its full timeout (default 5s) before retrying.

Mitigations that actually work: the single-request-reopen resolv.conf option, running a node-local DNS cache (NodeLocal DNSCache moves queries off conntrack entirely), or forcing TCP for DNS. On our managed clusters we deploy NodeLocal DNSCache once a cluster crosses roughly fifty nodes, because the conntrack race frequency scales with pod density.

A CoreDNS debugging checklist

A debugging checklist that has never let us down:

Read resolv.conf first. Exec into a pod with tools (kubectl run dbg --rm -it --image=busybox:1.36 -- sh) and read /etc/resolv.conf before anything else — half the mysteries are a webhook or dnsPolicy that rewrote it.
Query the Service IP directly. nslookup kubernetes.default.svc.cluster.local 10.96.0.10 separates "CoreDNS is broken" from "the path to CoreDNS is broken".
Query a CoreDNS pod IP directly. If the pod answers but the Service does not, suspect kube-proxy or the CNI, not DNS.
Turn on the log plugin in the Corefile for a few minutes and watch queries arrive — no log line means the packet never got there.
Check the forward target. On EKS the VPC resolver enforces a hard limit of 1024 packets per second per ENI, and a busy cluster can genuinely hit it.

CoreDNS vs external-dns — don't confuse them

One last distinction that saves arguments: CoreDNS answers queries inside the cluster; external-dns publishes records outside the cluster (Route 53, Cloudflare and friends) so the rest of the world can find your Services and Ingresses. They share nothing but the acronym. If nslookup works from a pod but your domain does not resolve from your laptop, the problem lives in external-dns or your zone delegation, not in CoreDNS — check the ownership TXT records external-dns writes next to each managed record before assuming the controller is broken.

DNS in Kubernetes rewards methodical debugging. Resolve the path one hop at a time — pod resolver, Service, CoreDNS pod, upstream — and the layer that lies to you becomes obvious within minutes.

Frequently asked questions

Why do my Kubernetes DNS lookups take exactly 5 seconds?

That's the resolver retry timer, not a slow server. A parallel A/AAAA UDP query pair hits a conntrack NAT insertion race and one packet is dropped; glibc waits its full 5-second timeout before retrying. Fix it with NodeLocal DNSCache, single-request-reopen, or TCP DNS.

What does ndots:5 do in Kubernetes?

It tells the resolver to append every search domain before trying a name that has fewer than five dots — which turns one external lookup into four to six. Use a trailing dot on hot external names or set a per-pod dnsConfig with ndots:2.

CoreDNS works but my domain won't resolve externally — why?

In-cluster resolution (CoreDNS) and external publishing (external-dns) are unrelated. If pods resolve fine but the public internet can't, check external-dns, the ownership TXT records, and your zone delegation. See DNS & domains.

How do I debug a DNS problem inside a pod?

Run a debug pod (kubectl run dbg --rm -it --image=busybox:1.36 -- sh), read /etc/resolv.conf, then query the CoreDNS Service IP and a CoreDNS pod IP directly to find which hop fails. See the troubleshooting guide.