Troubleshooting DNS Resolution Failures

Troubleshooting / Guide 97 of 103 / Intermediate

• Kubernetes1.36.4
• CoreDNSv1.14.2
• Cluster4 nodes
• CNICalico v3.32.1
• TimeAbout 30 min
• Reviewed21 August 2026

Written against the versions above. The broken-resolver Pod uses dnsConfig rather than damaging CoreDNS, so the cluster stays healthy while the failure is real.

Before you start

• The CoreDNS and service discovery guide, for the search path and ndots.
• A shell Pod with dig and nslookup. nicolaka/netshoot here.
• The Service has no endpoints guide, so you can tell a DNS failure from a Service failure.

1. 1

   NXDOMAIN means DNS answered

   Three lookups, and the middle one is the interesting failure.

   web-ok.t3.svc.cluster.local resolves. web-ok.default.svc.cluster.local gives NXDOMAIN, because there is no web-ok Service in the default namespace.

   The word to hold on to is answered. NXDOMAIN is a response: the query reached CoreDNS, CoreDNS looked, and reported that the name does not exist. So DNS is working perfectly and the name is wrong. That rules out CoreDNS, kube-proxy, the network and the CNI in one observation.

   The common causes of an NXDOMAIN on a name you expected to work:

   • Wrong namespace, as here. A short name only resolves within its own namespace.
   • The Service does not exist yet, or was deleted, or is named differently from the Deployment.
   • A typo in the name, including the namespace segment.

   The third lookup makes a related point. web-typo resolves to a ClusterIP even though that Service has no endpoints at all. DNS records exist for a Service the moment it has a ClusterIP, regardless of whether anything is behind it. So a successful lookup proves the Service object exists and nothing more; it says nothing about whether traffic will be served.

   dig states the same thing more precisely in its header. status: NOERROR for a name that exists, status: NXDOMAIN for one that does not. When comparing many names, dig +noall +comments | grep status is faster to read than nslookup.

   $kubectl exec client -n t3 -- nslookup -type=a web-ok.t3.svc.cluster.local 2>&1 | tail -3Name:	web-ok.t3.svc.cluster.localAddress: 10.101.223.82$kubectl exec client -n t3 -- nslookup -type=a web-ok.default.svc.cluster.local 2>&1 | tail -3** server can't find web-ok.default.svc.cluster.local: NXDOMAIN command terminated with exit code 1$kubectl exec client -n t3 -- nslookup -type=a web-typo 2>&1 | tail -3Name:	web-typo.t3.svc.cluster.localAddress: 10.97.101.148$kubectl exec client -n t3 -- dig +noall +comments web-ok.t3.svc.cluster.local A | grep status;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38746$kubectl exec client -n t3 -- dig +noall +comments nope.t3.svc.cluster.local A | grep status;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 39684

   Expected resultNOERROR, NXDOMAIN, and a resolvable name for an endpoint-less Service. All three answers are fast: NXDOMAIN is a reply, not a wait.

   Success conditionYou can produce both NOERROR and NXDOMAIN and say what each rules out.

2. 2

   A resolver that cannot be reached

   Now the other failure. dnsPolicy: None with dnsConfig lets a Pod be given a resolver of your choosing, and 10.255.255.254 is an address nothing answers on. That produces a genuine DNS outage inside one Pod without touching CoreDNS, so the rest of the cluster keeps working.

   /etc/resolv.conf confirms what the Pod was handed: one search domain and one dead nameserver. Note the cluster's usual four search domains and ndots:5 are absent, because dnsPolicy: None replaces the file entirely rather than adding to it.

   The failure looks completely different from NXDOMAIN:

   ;; communications error to 10.255.255.254#53: timed out
   ;; no servers could be reached

   Nothing answered. nslookup waits, retries, and eventually gives up; dig with +time=3 +tries=1 gives up in three seconds and says so plainly. On a real incident that difference in wording is the fastest way to know which failure you have:

   • NXDOMAIN - DNS answered. The name is wrong.
   • no servers could be reached / communications error ... timed out - nothing answered. The resolver, the route to it, or a policy blocking it.

   And the timing tells you too. An instant failure is an answer; a slow failure is silence.

   kubectl apply -f - <<'EOF'apiVersion: v1kind: Podmetadata:  name: baddns  namespace: t3spec:  dnsPolicy: None  dnsConfig:    nameservers: ["10.255.255.254"]    searches: ["t3.svc.cluster.local"]  containers:  - name: shell    image: nicolaka/netshoot:latest    command: ["sleep", "3600"]    resources: {requests: {cpu: 10m, memory: 16Mi}}EOFpod/baddns created$kubectl wait --for=condition=Ready pod/baddns -n t3 --timeout=240spod/baddns condition met$kubectl exec baddns -n t3 -- cat /etc/resolv.confsearch t3.svc.cluster.localnameserver 10.255.255.254$kubectl exec baddns -n t3 -- timeout 12 nslookup -type=a web-ok.t3.svc.cluster.local 2>&1 | tail -3; echo "exit=$?";; communications error to 10.255.255.254#53: timed out command terminated with exit code 1exit=0$kubectl exec baddns -n t3 -- timeout 12 dig +time=3 +tries=1 web-ok.t3.svc.cluster.local A 2>&1 | grep -E 'connection timed out|no servers|status';; no servers could be reached

   Expected resultA timeout rather than an answer. The Pod itself is perfectly healthy and Ready; only its name resolution is broken, which is exactly how this presents in production.

   Success conditionThe lookup times out instead of returning NXDOMAIN.

3. 3

   The one test that proves it is DNS

   This is the most useful two commands in the guide, and they settle the question conclusively.

   From the broken-DNS Pod, connecting by name fails:

   000 exit=28

   Connecting to the same Service's ClusterIP succeeds:

   200

   The Service is healthy. kube-proxy is healthy. The network is healthy. The only thing wrong is that this Pod cannot turn a name into that address.

   Run this pair whenever anything cannot reach anything. It splits the problem in half in seconds:

   • By-name fails, by-IP works -> DNS. Look at /etc/resolv.conf, CoreDNS, and any egress NetworkPolicy on UDP 53.
   • Both fail -> not DNS. The Service, the endpoints, the port, or a policy blocking the traffic itself.
   • By-name works, by-IP fails -> you used the wrong IP.

   One detail worth explaining, because it looks inconsistent with the previous guide. The by-name failure here is exit 28, a timeout, not exit 6 ("could not resolve host"). curl was given --max-time 6 and the dead resolver consumed all of it before curl could conclude that resolution had failed, so the overall time limit expired first. Shorten the DNS timeout or lengthen --max-time and you get 6 instead.

   That is a useful caution about exit codes generally: they tell you what curl gave up on, which is not always the underlying cause. The by-IP comparison has no such ambiguity, which is why it is the better test.

   $kubectl exec baddns -n t3 -- curl -s -o /dev/null -w '%{http_code} exit=' --max-time 6 http://web-ok; echo $?000 exit=command terminated with exit code 2828$kubectl exec baddns -n t3 -- curl -s -o /dev/null -w '%{http_code}\n' --max-time 6 http://10.101.223.82200

   Expected resultFailure by name, success by address, from the same Pod seconds apart. That is the proof; nothing else needs to be checked to establish that DNS is the fault.

   Success conditionThe ClusterIP works while the name does not.

Troubleshooting

• Nothing in the cluster can resolve anything.

  Why: CoreDNS is down, or the kube-dns Service is not routing to it. DNS is reached through a Service, so kube-proxy is in the path.

  Fix:Work down the chain. kubectl get pods -n kube-system -l k8s-app=kube-dns, then kubectl get endpointslice -n kube-system -l kubernetes.io/service-name=kube-dns. If both look right, query a CoreDNS Pod IP directly: kubectl exec <pod> -- dig @<coredns-pod-ip> kubernetes.default.svc.cluster.local. Working against the Pod IP but not 10.96.0.10 puts the fault in kube-proxy, not DNS.

• One Pod cannot resolve and every other Pod can.

  Why: That Pod's own resolver configuration, or a NetworkPolicy selecting only it.

  Fix:Read its file first: kubectl exec <pod> -- cat /etc/resolv.conf. A nameserver that is not the kube-dns ClusterIP means dnsPolicy is the cause: Default inherits the node's resolver and None replaces the file, as in step 2. If the file is correct, check for an egress NetworkPolicy: any egress policy selecting the Pod blocks UDP 53 unless it explicitly allows it, which is the single most common cause of this symptom.

• In-cluster names resolve and external names do not.

  Why: CoreDNS forwards anything outside the cluster domain to the node's resolver, and that path is broken.

  Fix:The split is the diagnosis: the kubernetes plugin works and forwarding does not. Check the node's own DNS. On a systemd-resolved host the node's /etc/resolv.conf may point at 127.0.0.53, which means nothing to CoreDNS in a Pod; the kubelet's --resolv-conf should name the real file, usually /run/systemd/resolve/resolv.conf.

• A hostNetwork Pod cannot resolve cluster names.

  Why: hostNetwork: true implies the node's resolver unless you say otherwise, so the Pod never gets the cluster nameserver.

  Fix:Set dnsPolicy: ClusterFirstWithHostNet. Confirm by reading /etc/resolv.conf in the Pod: if the nameserver is not the kube-dns ClusterIP, the policy is why.

• Resolution works but is intermittently slow.

  Why: Usually ndots:5 multiplying every short name by the number of search domains, sometimes conntrack table pressure on UDP.

  Fix:Use fully qualified names with a trailing dot in hot paths: svc.ns.svc.cluster.local. is one query instead of five. The CoreDNS guide measures this. For a large cluster, NodeLocal DNSCache removes the per-query round trip entirely.

• A name resolves to an address that does not work.

  Why: The Service exists and has no usable endpoints. DNS records are created for any Service with a ClusterIP, whether or not anything is behind it.

  Fix:As step 1 shows, a successful lookup proves only that the Service object exists. Check the EndpointSlice next: kubectl get endpointslice -l kubernetes.io/service-name=<svc>. This is why "DNS works" is never sufficient evidence that a Service is healthy.

Official sources

• Debugging DNS resolution
• DNS for Services and Pods
• Pod DNS config
• Customizing DNS Service
• NodeLocal DNSCache