CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Troubleshooting a Service With No Endpoints

Two broken Services with identical symptoms. One has an empty EndpointSlice from a one-letter typo; the other has two healthy endpoints and still refuses every connection. Checking the slice first is what separates them.

Troubleshooting Guide 98 of 103 Intermediate

Written against the versions above. The curl exit codes are the fastest signal here and are worth learning: 7 is refused, 28 is a timeout.

Any cluster does. A netshoot Pod is used as the client throughout.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. A one-letter typo, and an empty slice

    The Service selects app: webb. The Deployment's Pods are labelled app: web. One letter.

    The Service is created without complaint, gets a ClusterIP, and looks entirely normal in kubectl get svc. Nothing anywhere reports a problem, because nothing is wrong with the object: a selector that matches nothing is a valid selector.

    The EndpointSlice is where it shows:

    web-typo-j8ncx endpoints=null

    And curl fails with exit code 7, connection refused. Not a timeout: DNS resolved, the ClusterIP was reached, and there was no rule to rewrite the packet to, so it was refused immediately.

    The two-command diagnosis, and this is the whole method:

    kubectl get svc <name> -o jsonpath='{.spec.selector}'
kubectl get pods -l <that selector>

    selector={"app":"webb"} then No resources found. Conclusive, in under ten seconds, with no guessing.

    kubectl describe svc shows the same thing more compactly: Selector: app=webb above an empty Endpoints: line. Reading those two lines together is the habit worth building.

    The usual causes, in rough order of frequency: a typo; labels placed on the Deployment's metadata rather than its spec.template.metadata so the Pods never get them; and a Service in a different namespace from its Pods, since a selector never crosses namespaces.

    bash Example session 
    kubectl get pods -n t3 --show-labels --no-headers | awk '{print $1, $6}'client <none>web-7fbc579fd4-rxn8q app=web,pod-template-hash=7fbc579fd4web-7fbc579fd4-xr86g app=web,pod-template-hash=7fbc579fd4kubectl apply -f - <<'EOF'apiVersion: v1kind: Servicemetadata:  name: web-typo  namespace: t3spec:  selector:    app: webb  ports:  - {port: 80, targetPort: 80}EOFservice/web-typo createdsleep 10; kubectl get svc web-typo -n t3NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGEweb-typo   ClusterIP   10.97.101.148   <none>        80/TCP    10skubectl get endpointslice -n t3 -l kubernetes.io/service-name=web-typo -o jsonpath="{range .items[*]}{.metadata.name}{\" endpoints=\"}{.endpoints}{\"\n\"}{end}"web-typo-j8ncx endpoints=nullkubectl exec client -n t3 -- curl -s -o /dev/null -w '%{http_code} exit=' --max-time 5 http://web-typo; echo $?000 exit=command terminated with exit code 77kubectl get svc web-typo -n t3 -o jsonpath="selector={.spec.selector}{\"\n\"}"selector={"app":"webb"}kubectl get pods -n t3 -l app=webb --no-headers 2>&1 | tail -1No resources found in t3 namespace.kubectl describe svc web-typo -n t3 | grep -E 'Selector|Endpoints'Selector:                 app=webbEndpoints:

    Expected resultendpoints=null, exit 7, and the selector matching nothing. Note pod-template-hash in the labels: that is added by the ReplicaSet, and selecting on it works but breaks on the next rollout.

    Success conditionkubectl get pods -l <selector> returns nothing for the Service's own selector.

  2. Endpoints present, connection still refused

    Now the case that catches people who learned the previous step and stopped there. This Service's selector is correct, app: web, and the EndpointSlice proves it:

    ["10.244.93.63"] ["10.244.100.44"] port=8080

    Two healthy endpoints. And curl still fails with exit 7, identical to the typo case.

    The difference is in that same output: port=8080. The Service's targetPort is 8080, and nginx listens on 80. The endpoints are correct, kube-proxy's rules are correct, and the packet is being delivered to a port where nothing is listening. The container refuses it, which is why the failure is a refusal rather than a timeout.

    So the two cases are told apart by the slice, not by the symptom:

    • Empty slice -> the selector is wrong.
    • Populated slice, still refused -> the port is wrong, or the container is not listening on the port it should be.

    The last command is worth its own note. .spec.containers[0].ports returns empty, because this Deployment was created with kubectl create deployment and declares no containerPort at all.

    That is not the bug. containerPort is documentation: it does not open, publish or restrict anything, and a Service's targetPort can name a port the container never declared. It works if the process is listening and fails if it is not, regardless of what the spec says.

    Which means the spec cannot answer the question. To find what a container is really listening on, ask it: kubectl exec -- netstat -tlnp, or test the Pod IP and port directly from another Pod.

    bash Example session 
    kubectl apply -f - <<'EOF'apiVersion: v1kind: Servicemetadata:  name: web-badport  namespace: t3spec:  selector:    app: web  ports:  - {port: 80, targetPort: 8080}EOFservice/web-badport createdsleep 10; kubectl get endpointslice -n t3 -l kubernetes.io/service-name=web-badport -o jsonpath="{range .items[*]}{.endpoints[*].addresses}{\" port=\"}{.ports[*].port}{\"\n\"}{end}"["10.244.93.63"] ["10.244.100.44"] port=8080kubectl exec client -n t3 -- curl -s -o /dev/null -w '%{http_code} exit=' --max-time 5 http://web-badport; echo $?000 exit=command terminated with exit code 77kubectl get svc web-badport -n t3 -o jsonpath="port={.spec.ports[0].port}{\" targetPort=\"}{.spec.ports[0].targetPort}{\"\n\"}"port=80 targetPort=8080kubectl get pod -n t3 -l app=web -o jsonpath="{.items[0].spec.containers[0].ports}{\"\n\"}"

    Expected resultTwo endpoints and a refused connection. The empty ports output is real: the container declares none, which changes nothing about what it listens on.

    Success conditionThe slice is populated and the connection still fails.

  3. The same Service done right

    kubectl expose --target-port=80 produces the working version, and it is worth seeing the healthy output so the broken ones have something to be compared against.

    Two endpoints, both ready=true, and a 200.

    So the full diagnostic order for a Service that will not serve, in the order that eliminates the most per command:

    1. kubectl get endpointslice -l kubernetes.io/service-name= - empty means the selector, populated means look further.
    2. Read the port from the slice - that is what traffic is being sent to, and comparing it against what the container listens on settles the second case.
    3. Test the Pod IP directly - curl : from another Pod. Working here and failing through the Service points at kube-proxy; failing both ways points at the container.
    4. Check readiness - endpoints marked ready=false are excluded from load balancing, so a Service with only unready endpoints behaves exactly like one with none.

    And the exit codes, which are the cheapest signal available:

    • 7 - connection refused. No endpoint, wrong port, or nothing listening.
    • 28 - timeout. Packets are being dropped: NetworkPolicy, a firewall, or a broken overlay.
    • 6 - could not resolve the name. A DNS problem, not a Service problem.

    That last distinction matters more than it looks. Exit 6 or a DNS timeout means the Service may be perfectly healthy and the client simply cannot find it, which is a different guide.

    bash Example session 
    kubectl expose deployment web -n t3 --name=web-ok --port=80 --target-port=80service/web-ok exposedsleep 10; kubectl get endpointslice -n t3 -l kubernetes.io/service-name=web-ok -o jsonpath="{range .items[*].endpoints[*]}{.addresses}{\" ready=\"}{.conditions.ready}{\"\n\"}{end}"["10.244.93.63"] ready=true["10.244.100.44"] ready=truekubectl exec client -n t3 -- curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 http://web-ok200

    Expected resultThe same two Pod IPs as the broken Service had, now with the right port, returning 200. The endpoints were never the problem in step 2.

    Success conditioncurl through the corrected Service returns 200.

Troubleshooting

Official sources

← All CKA guides