CertGrid CertGrid
Troubleshooting·Kubernetes and Cloud Native Security Associate

Cluster DNS and Service Discovery

A Pod reported NXDOMAIN for the apiserver and then fetched a page from it seconds later. This guide takes that apart, because the wrong conclusion - that the Pod is isolated from the API - would have been a security finding that was not true.

Cloud Native Security Guide 5 of 42 Beginner

Written against the versions above. CoreDNS at 10.96.0.10 with the standard kubeadm search list and `ndots:5`. busybox 1.37's `nslookup` is the tool that misleads here; it behaves this way on every Kubernetes cluster.

The cka5001 cluster: one control plane and 3 schedulable workers, on Cilium.
Server NameIP AddressOSRolesCPURAMHDD
CKA5001192.168.0.41Ubuntu 26.04 LTSControl Plane Node (tainted NoSchedule)2 Core4 GB50 GB
CKA5001-NODE01192.168.0.42Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA5001-NODE02192.168.0.43Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA5001-NODE03192.168.0.44Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. The failure, exactly as it appeared

    One name, the short one everybody uses.

    bash Example session 
    kubectl run kcsa-dns --image=docker.io/library/busybox:1.37 --restart=Never --command -- sh -c 'sleep 300' >/dev/null && kubectl wait --for=condition=Ready pod/kcsa-dns --timeout=120s >/dev/null && kubectl exec kcsa-dns -- nslookup kubernetes.default.svc 2>&1 | grep -vE '^$' | head -4; echo "--- NXDOMAIN, and a non-zero exit"Server:		10.96.0.10Address:	10.96.0.10:53** server can't find kubernetes.default.svc: NXDOMAIN** server can't find kubernetes.default.svc: NXDOMAIN--- NXDOMAIN, and a non-zero exit

    Expected result** server can't find kubernetes.default.svc: NXDOMAIN and command terminated with exit code 1.

    Success conditionYou have the failure in front of you.

  2. The same name, fully qualified

    Add the suffix the search list would have added.

    bash Example session 
    kubectl exec kcsa-dns -- nslookup kubernetes.default.svc.cluster.local 2>&1 | grep -vE '^$' | tail -3; echo "--- the SAME name fully qualified resolves, so DNS was never broken"Address:	10.96.0.10:53Name:	kubernetes.default.svc.cluster.localAddress: 10.96.0.1--- the SAME name fully qualified resolves, so DNS was never brokenkubectl exec kcsa-dns -- cat /etc/resolv.conf; echo "--- the search list nslookup ignored and every other tool uses"search default.svc.cluster.local svc.cluster.local cluster.local practicelabpro.localnameserver 10.96.0.10options ndots:5--- the search list nslookup ignored and every other tool uses

    Expected resultName: kubernetes.default.svc.cluster.local, Address: 10.96.0.1. Then a search list of four domains, nameserver 10.96.0.10 and options ndots:5.

    Success conditionYou know the resolver configuration is correct.

  3. The tool that gets it right, on the same Pod

    Nothing changed except which binary asked.

    bash Example session 
    kubectl exec kcsa-dns -- sh -c 'wget -q -O- --no-check-certificate --timeout=5 https://kubernetes.default.svc/version 2>&1 | head -c 130'; echo; echo "--- wget applies the search list, reaches the apiserver by SHORT name, and gets a real answer"{  "major": "1",  "minor": "36",  "emulationMajor": "1",  "emulationMinor": "36",  "minCompatibilityMajor": "1",  "minCompat--- wget applies the search list, reaches the apiserver by SHORT name, and gets a real answerkubectl delete pod kcsa-dns --wait=true 2>&1 | tail -1; kubectl get pod kcsa-dns 2>&1 | tail -1pod "kcsa-dns" deleted from default namespaceError from server (NotFound): pods "kcsa-dns" not found

    Expected resultThe apiserver's /version payload - "major": "1", "minor": "36" - fetched by the SHORT name. Then the Pod is deleted.

    Success conditionYou can tell a resolver problem from a tool problem.

Troubleshooting

Official sources

← All KCSA guides