CoreDNS in Production: Scaling, Tuning, and Debugging Kubernetes DNS

Every Kubernetes service discovery call goes through CoreDNS. When you write http://payments-service/api, the application sends a DNS query that CoreDNS answers with the ClusterIP. At low cluster scale, this is invisible — CoreDNS just works. At production scale with thousands of pods making hundreds of DNS queries per second, CoreDNS becomes a hot-path dependency that requires deliberate capacity management.

The most insidious CoreDNS problem is the one you don't see coming: the default Kubernetes DNS configuration causes each short service name to generate up to 5 DNS queries before it gets an answer. A microservice making 10 inter-service calls per request is generating 50 DNS queries per request — and most of them return NXDOMAIN before the correct record is found. This doesn't break anything visibly; it just adds latency and load that's hard to attribute without knowing where to look.

How Kubernetes DNS Works

CoreDNS runs as a Deployment in kube-system, exposed via a ClusterIP Service (typically 10.96.0.10 or similar). The kubelet configures every pod's /etc/resolv.conf to point to this ClusterIP:

nameserver 10.96.0.10
search production.svc.cluster.local svc.cluster.local cluster.local us-east-1.compute.internal
options ndots:5

The search list: When a pod queries payments-service, the DNS resolver tries the query with each search domain appended, in order:

1. payments-service.production.svc.cluster.local — found (the correct ClusterIP)
2. (stops here)

But for external names like api.github.com, with ndots:5:

1. api.github.com.production.svc.cluster.local — NXDOMAIN
2. api.github.com.svc.cluster.local — NXDOMAIN
3. api.github.com.cluster.local — NXDOMAIN
4. api.github.com.us-east-1.compute.internal — NXDOMAIN
5. api.github.com. — returned (correct, but 5 queries before getting here)

ndots:5 means any name with fewer than 5 dots gets the search list treatment. api.github.com has 2 dots — all 5 search domains are tried before the absolute name is queried.

DNS Policy

dnsPolicy controls how a pod's /etc/resolv.conf is configured:

The ndots Problem

For workloads that make many external API calls (payment processors, telemetry APIs, external services), the 5-search-domain expansion adds measurable latency. Each NXDOMAIN requires a full round-trip to CoreDNS.

Fix 1: Use fully-qualified domain names (FQDN) with trailing dot

# In application config or environment variables
API_ENDPOINT: "https://api.github.com."   # Trailing dot = absolute name, skips search list

The trailing dot tells the resolver this is a fully-qualified name — skip the search domain expansion entirely.

Fix 2: Reduce ndots via pod dnsConfig

1spec:
2  dnsConfig:
3    options:
4      - name: ndots
5        value: "2"   # Only names with <2 dots get search list treatment
6  # With ndots:2, api.github.com has 2 dots — queried as absolute immediately
7  # kubernetes.default still works (1 dot < 2 → search list applied → found)

Fix 3: ndots at namespace or cluster level via LimitRange / admission webhook

Enforcing ndots: 2 cluster-wide requires a mutating webhook. Alternatively, add it to your base container configuration via Helm chart defaults or Kyverno mutation.

The DNS Race Condition

A subtler problem: Kubernetes pods often have two DNS query types fire simultaneously for the same name — an A record query and an AAAA (IPv6) record query. Both queries are sent over the same UDP socket to CoreDNS.

When these concurrent queries arrive at a connection-tracking NAT table (common in iptables-based Kubernetes networking), they share the same source port. The conntrack entry collision causes one query to be dropped — the pod retries, adding 5 seconds of latency (the DNS timeout).

Symptoms: Intermittent 5-second delays on service calls with no apparent pattern. DNS resolution sometimes takes 5+ seconds even when CoreDNS is healthy.

Fix 1: single-request-reopen (Linux resolver option)

spec:
  dnsConfig:
    options:
      - name: single-request-reopen
        # Opens a new socket for the A query if the AAAA query is using the same 5-tuple
        # Effectively serialises A and AAAA queries to avoid the conntrack race

This is the recommended fix for most clusters. It serialises A and AAAA queries instead of sending them in parallel, eliminating the race condition.

Fix 2: single-request (stronger serialization)

spec:
  dnsConfig:
    options:
      - name: single-request
        # Only sends one query at a time (A first, then AAAA if needed)

Fix 3: NodeLocal DNSCache (best long-term solution — see below)

NodeLocal DNSCache caches DNS responses on each node and uses a dedicated link-local IP, bypassing the conntrack NAT path entirely. This eliminates the race condition at the infrastructure level.

The Corefile: CoreDNS Configuration

CoreDNS configuration lives in a ConfigMap:

kubectl get configmap coredns -n kube-system -o yaml

Default Corefile:

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

Tuning for Production

.:53 {
    errors
    health {
       lameduck 5s
    }
    ready

    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }

    # Serve /etc/hosts from ConfigMap instead of node /etc/hosts
    # hosts /etc/coredns/NodeHosts {
    #   fallthrough
    # }

    prometheus :9153

    forward . /etc/resolv.conf {
       max_concurrent 1000
       # UDP is the default for upstream queries; use force_tcp to force TCP
    }

    cache 30 {
       success 9984 30   # Cache up to 9984 successful responses for 30s
       denial 9984 5     # Cache NXDOMAIN responses for 5s (prevent thundering herd)
    }

    loop
    reload
    loadbalance
}

Enabling query logging (for debugging only — high overhead):

.:53 {
    log   # Add this line to log all queries to stdout
    ...
}

Enable logging temporarily via ConfigMap edit — CoreDNS reloads automatically (the reload plugin polls for changes).

Scaling CoreDNS

Default EKS clusters ship with 2 CoreDNS replicas. Insufficient replicas cause CoreDNS CPU saturation, which manifests as increased DNS latency across the entire cluster.

Horizontal Pod Autoscaler

1apiVersion: autoscaling/v2
2kind: HorizontalPodAutoscaler
3metadata:
4  name: coredns
5  namespace: kube-system
6spec:
7  scaleTargetRef:
8    apiVersion: apps/v1
9    kind: Deployment
10    name: coredns
11  minReplicas: 2
12  maxReplicas: 10
13  metrics:
14    - type: Resource
15      resource:
16        name: cpu
17        target:
18          type: Utilization
19          averageUtilization: 70

Cluster Proportional Autoscaler

For clusters where node count changes frequently, the cluster-proportional-autoscaler scales CoreDNS based on cluster size:

1# Rule: 1 CoreDNS replica per 8 nodes, minimum 2, maximum 15
2ladder:
3  coresToReplicas:
4    - [1, 2]
5  nodesToReplicas:
6    - [8, 2]    # 0-7 nodes: 2 replicas
7    - [16, 3]   # 8-15 nodes: 3 replicas
8    - [24, 4]   # 16-23 nodes: 4 replicas
9    - [100, 10] # 24-99 nodes: 10 replicas

EKS can manage CoreDNS scaling via add-on configuration, but the cluster-proportional autoscaler is not enabled by default — you need to configure it explicitly or use the coredns managed add-on with autoscaling settings.

Resource Sizing

CoreDNS pod resources for a 50-100 node cluster:

1resources:
2  requests:
3    cpu: 100m
4    memory: 70Mi
5  limits:
6    cpu: 500m
7    memory: 170Mi

Monitor coredns_dns_requests_total and coredns_dns_request_duration_seconds — if P99 latency exceeds 10ms under normal load, add replicas.

NodeLocal DNSCache

NodeLocal DNSCache runs a DNS caching agent (based on CoreDNS) as a DaemonSet on each node. The agent listens on 169.254.20.10 — a link-local address that only exists on the node. The kubelet configures pods to use this address for DNS (on clusters where NodeLocal DNSCache is deployed).

Benefits:

• Eliminates the conntrack race condition (link-local traffic bypasses NAT)
• Sub-millisecond cache hits (on-node, no network hop)
• Dramatically reduces load on CoreDNS pods

1# Deploy NodeLocal DNSCache (replace PILLAR_DNS_SERVER with your cluster's kube-dns ClusterIP)
2PILLAR_DNS_SERVER=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
3
4# Pin to your cluster's Kubernetes version — avoid the master branch in production
5curl -Lo nodelocaldns.yaml \
6  "https://raw.githubusercontent.com/kubernetes/kubernetes/v1.29.0/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml"
7
8sed -i "s/__PILLAR__DNS__SERVER__/${PILLAR_DNS_SERVER}/g" nodelocaldns.yaml
9sed -i "s/__PILLAR__LOCAL__DNS__/169.254.20.10/g" nodelocaldns.yaml
10sed -i "s/__PILLAR__DNS__DOMAIN__/cluster.local/g" nodelocaldns.yaml
11
12kubectl apply -f nodelocaldns.yaml

On EKS, NodeLocal DNSCache is available as an add-on or can be deployed manually. After deployment, new pods automatically receive 169.254.20.10 as their nameserver if you configure the kubelet to use NodeLocal DNSCache (on EKS, this is handled by the add-on setup).

Debugging DNS

1# Run a pod with DNS diagnostic tools
2kubectl run dnsutils \
3  --image=registry.k8s.io/e2e-test-images/agnhost:2.39 \
4  --restart=Never \
5  -it -- bash
6
7# Inside the pod:
8# Check /etc/resolv.conf
9cat /etc/resolv.conf
10
11# Test internal service resolution
12nslookup kubernetes.default
13nslookup my-service.production.svc.cluster.local
14
15# Test external resolution
16nslookup api.github.com
17
18# Test with specific nameserver
19dig @10.96.0.10 my-service.production.svc.cluster.local
20
21# Measure DNS latency
22for i in {1..10}; do
23  time nslookup my-service.production.svc.cluster.local
24done

1# Check CoreDNS logs for errors
2kubectl logs -n kube-system -l k8s-app=kube-dns --tail=100 -f
3
4# CoreDNS metrics
5kubectl port-forward -n kube-system svc/kube-dns 9153:9153
6curl http://localhost:9153/metrics | grep coredns_dns_request
7
8# Check CoreDNS pod resource usage
9kubectl top pods -n kube-system -l k8s-app=kube-dns

For diagnosing whether DNS is the root cause of a service call failure:

# Add strace to capture DNS syscalls (if debug container available)
kubectl debug -it my-pod --image=nicolaka/netshoot -- strace -e trace=network \
  curl http://my-service/api 2>&1 | grep -E "connect|recvfrom"

Frequently Asked Questions

Why are some DNS queries taking exactly 5 seconds?

Five seconds is the Linux DNS resolver's retry timeout. It fires when a DNS query is dropped (no response received) — the resolver waits 5 seconds before retrying. This is almost always the conntrack race condition causing a UDP packet drop. Fix with single-request-reopen in dnsConfig options, or deploy NodeLocal DNSCache.

How do I make CoreDNS resolve custom domain names?

Add a custom zone to the Corefile using the rewrite or hosts plugin:

.:53 {
    ...
    rewrite name my-internal-service.corp internal-service.production.svc.cluster.local
    ...
}

Or use the forward plugin to delegate a custom TLD to an internal DNS server:

.:53 {
    ...
    forward corp.example.com 10.0.0.53   # Internal DNS server for .corp.example.com
    ...
}

Can I use CoreDNS with Istio service mesh?

Yes. Istio's service discovery uses Kubernetes DNS for initial resolution, then Envoy intercepts the connection for routing and mTLS. CoreDNS sees the DNS query; Envoy intercepts the resulting TCP connection. The ndots and single-request-reopen fixes apply equally in Istio-meshed pods. For services using Istio's ServiceEntry for external service access, CoreDNS is not involved — Envoy resolves the external address directly.

For CoreDNS Corefile tuning and advanced configuration — autopath, negative caching, custom forward zones, and ndots optimization — see Kubernetes DNS: CoreDNS Configuration and Tuning. For network-level security policies that apply after DNS resolution, see Kubernetes NetworkPolicy Patterns. For EKS-specific DNS configuration including NodeLocal DNSCache setup on EKS, see EKS Networking Deep Dive. For Cilium's FQDN-based network policies that rely on DNS interception, see Cilium eBPF Kubernetes Networking.

Diagnosing intermittent DNS failures or slow service discovery in a production cluster? Talk to us at Coding Protocols — we help platform teams identify and fix DNS bottlenecks before they become service reliability incidents.