About the DNS Outage Playbook
DNS failures rarely present as DNS failures. They present as one service being unreachable, as intermittent timeouts, or as an application that started working again by itself — because a resolver moved to a different upstream or a cached record expired. The first job in a suspected DNS incident is to establish which resolver answered and whether the answer was cached.
This playbook works down the resolution path in order: the application's own resolver configuration, the local stub or node resolver, the cluster DNS service, the upstream forwarders, and finally the authoritative servers. Each step has a check that distinguishes a real failure from a stale cached answer.
The specific trap in Kubernetes is that most DNS incidents there are not DNS at all. They are a NetworkPolicy blocking egress on port 53, an ndots setting turning one lookup into five, or a CoreDNS pod that lost its upstream. All three look like resolution failure from inside the pod.
Frequently asked questions
How do I tell whether a DNS answer came from cache?
Compare the TTL in successive queries. A cached record counts down between queries; a fresh authoritative answer returns the full TTL each time. Query the authoritative nameserver directly to bypass every cache — if authoritative and resolver answers differ, you have a caching or propagation problem, not a record problem.
Why does DNS work from the node but not from inside a pod?
Pods resolve through the cluster DNS service, not the node's resolver, so the two paths share almost nothing. The usual causes are a NetworkPolicy that denies egress without allowing UDP and TCP port 53 to kube-dns, CoreDNS pods that are unhealthy or scheduled onto a broken node, or a ConfigMap change that removed the forwarder for external names.
What is the ndots problem?
Kubernetes sets ndots to 5 by default, so any name with fewer than five dots is first tried against every entry in the search list. Resolving an external hostname can mean four failed lookups before the correct one, which multiplies DNS load and adds latency to every connection. Appending a trailing dot makes a name fully qualified and skips the search list entirely.
How long does a DNS change take to propagate?
At most the TTL of the old record, assuming every resolver honours it — and some do not, particularly consumer ISP resolvers that impose their own minimums. The practical procedure is to lower the TTL well before the change, wait for the old TTL to expire everywhere, make the change, verify, then raise the TTL again.
Should I run a DNS cache on each node?
It is usually worth it at scale. NodeLocal DNSCache puts a caching resolver on every node, which cuts latency, removes a large share of load from the central DNS pods, and avoids a known conntrack race on UDP that causes intermittent lookup failures under load. The cost is another component to operate and to remember during incidents.
Need this managed for you, not just automated?
We're also a hands-on DevOps consultancy — Kubernetes, CI/CD, and cloud infrastructure.