CertGrid CertGrid
Troubleshooting·Certified Kubernetes Administrator

Troubleshooting Ingress 404 and 502 Errors

The status code tells you which hop failed, and that is most of the diagnosis. 404 means nothing matched, 503 means matched but no backend, 502 means the backend answered badly. The trap is a 404 the controller never generated.

Troubleshooting Guide 100 of 103 Intermediate

Written against the versions above. Status codes and the access log format are ingress-nginx behaviour. The reasoning transfers to other controllers; the exact bodies and log lines do not.

Requests enter on the controller's HTTP node port, 30561 in this install.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. Three codes, three different hops

    An Ingress request crosses four hops, and each failure has its own code. Learning which hop each code implicates removes most of the guesswork:

    • 404 - the controller found no rule matching this host and path. The request never reached a backend.
    • 503 - a rule matched, but the Service behind it has no ready endpoints. The controller had nowhere to send the request.
    • 502 - the request reached a Pod and the Pod's answer was unusable: connection refused, connection reset, a malformed response, or the wrong protocol.
    • 504 - the Pod accepted the connection and did not answer in time. Usually a slow application, occasionally a timeout set too low.

    The distinction that matters most is 503 against 502. A 503 means the controller never opened a connection to anything, so the problem is in the Service, the selector, or the readiness probe. A 502 means it did connect and something went wrong at the far end, so the problem is in the Pod, its port, or its protocol. They feel similar and they lead to opposite investigations.

    One trap outranks all of this, and it is the subject of step 2: a 404 the controller never produced.

    bash Example session 
    curl -s -o /dev/null -w 'right host: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/right host: 200curl -s -o /dev/null -w 'wrong host: %{http_code}\n' --max-time 8 -H 'Host: other.cg.test' http://127.0.0.1:30561/blue/wrong host: 404curl -s -o /dev/null -w 'unmatched path: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/nothing/unmatched path: 404

    Expected resultA working request, then two ways to miss. Both misses return the same 404, which is why the code alone does not tell you whether the host or the path was wrong.

    Success conditionYou can produce a 404 from a host miss and from a path miss.

  2. The trap: whose 404 is it?

    This is the failure that wastes the most time, so it is worth doing slowly.

    An Ingress with path: /blue and pathType: Prefix pointing at a healthy backend returns:

    /blue/ -> 404

    The obvious conclusion is that the rule is not matching. It is wrong. Look at the body:

    <head><title>404 Not Found</title></head>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.29.8</center>

    That footer is the giveaway. nginx/1.29.8 is the backend's version, not the controller's. The controller matched the rule perfectly, forwarded /blue/ unchanged, and the backend has no /blue/ directory, so the backend generated the 404 and the controller passed it back untouched.

    So the first question for any Ingress 404 is not "why is my rule not matching". It is who wrote this response. Three ways to answer it:

    Read the body. A controller 404 and a backend 404 look different. ingress-nginx's own 404 body has no server footer; the backend's carries its version. Any distinguishing content in the backend's error pages works the same way.

    Ask the backend directly, bypassing the Ingress entirely. A Pod in the same namespace requesting the same path through the Service:

    wget: server returned error: HTTP/1.1 404 Not Found

    The backend returns 404 for that path with no Ingress involved. That settles it, and it is the most conclusive of the three because it removes the controller from the picture.

    Read the access log (step 4). A request that reached the controller appears there; one that did not, does not.

    The fix in this case is a rewrite-target annotation, which the routing rules guide covers. The point here is the diagnosis: a 404 with a backend's fingerprint on it is a path problem inside the application, not a rule problem in the Ingress.

    bash Example session 
    curl -s -o /dev/null -w '/blue/ -> %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue//blue/ -> 404curl -s --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/ | grep -iE 'title|center'<head><title>404 Not Found</title></head><center><h1>404 Not Found</h1></center><hr><center>nginx/1.29.8</center>kubectl logs t -n ing 2>&1 | tail -2wget: server returned error: HTTP/1.1 404 Not Found

    Expected resultA 404 whose body names the backend, and the same 404 reproduced with the Ingress out of the path. The t Pod ran wget -qO- http://blue/blue/, going straight to the Service.

    Success conditionYou can tell a controller 404 from a backend 404, and confirm it by bypassing the Ingress.

  3. 503 is an endpoints problem, and it says so

    Scale the backend to zero and request the path that worked a second ago:

    green endpoints=null
green, no endpoints: 503
blue, healthy: 200

    The endpoints=null and the 503 are the same fact stated twice. The Service still exists, the Ingress rule still matches, and the EndpointSlice has no addresses, so there is nothing to forward to. The controller answers 503 itself.

    Note that blue still returns 200. A 503 on one path while others work is normal and it narrows the search immediately: whatever is wrong is specific to that Service and its Pods, not to the controller or the Ingress.

    Scaling to zero is the deliberate version. The ones you meet by accident all reduce to the same missing endpoints:

    • The readiness probe is failing. Pods are Running, and not Ready, so they are excluded from the EndpointSlice. kubectl get pods shows 1/1 or 0/1 in the READY column, and that column is the answer.
    • The Service selector does not match the Pod labels. The most common cause, and the most invisible, because both objects look fine on their own. kubectl get endpointslice -l kubernetes.io/service-name= returning nothing proves it.
    • The Ingress names a port the Service does not define. A port: number: 8080 against a Service listening on 80 is a valid Ingress and a permanent 503.
    • The Ingress and the Service are in different namespaces. An Ingress can only reference Services in its own namespace, and a Service that does not exist there is treated as one with no endpoints.
    • All Pods were evicted or are Pending. Then the endpoints question is a scheduling question.

    The check that covers every one of these is a single command:

    kubectl get endpointslice -n <ns> -l kubernetes.io/service-name=<svc>

    Addresses listed means the problem is downstream of the Service. Nothing listed means the problem is upstream of it, and 503 is the correct answer to a request the controller cannot route.

    bash Example session 
    kubectl scale deploy green -n ing --replicas=0deployment.apps/green scaledsleep 14; kubectl get endpointslice -n ing -l kubernetes.io/service-name=green -o jsonpath="green endpoints={.items[0].endpoints}{\"\n\"}"green endpoints=nullcurl -s -o /dev/null -w 'green, no endpoints: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/green/green, no endpoints: 503curl -s -o /dev/null -w 'blue, healthy: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/blue, healthy: 200

    Expected resultNo endpoints and a 503 on that path, with the other path unaffected. The Ingress and the Service were not touched.

    Success conditionYou can produce a 503 by removing endpoints, and confirm the cause from the EndpointSlice.

  4. The access log is the record of everything above

    The controller logs every request it handled, and reading that log is faster than reproducing failures one at a time:

    "GET /green/ HTTP/1.1" 200
"GET /blue/ HTTP/1.1" 200
"GET /nothing/ HTTP/1.1" 404
"GET /green/ HTTP/1.1" 503
"GET /blue/ HTTP/1.1" 200

    Five lines and the whole session is in them: both paths healthy, the unmatched path returning 404, then green going 503 after the scale-down while blue stayed 200. The grep -oE in the command trims the full log lines down to method, path and status; drop it to see client addresses, upstream response times and the upstream address actually chosen, all of which matter for a 502 or a 504.

    Two things to know about reading it.

    Absence is evidence. If a request that failed for the client does not appear in the log at all, it never reached the controller. Then the problem is in front of the controller: DNS, the load balancer, the node port, a NetworkPolicy, or the client itself. That single distinction splits an Ingress problem cleanly into two halves, and it is the most useful thing the log does.

    Read the right replica. kubectl logs deploy/ingress-nginx-controller follows one Pod. With several controller replicas, a given request went through exactly one of them, so a failure you cannot find may be in another replica's log. kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller --tail=50 covers all of them.

    The controller's own logs, as opposed to its access log, are where configuration problems surface: a TLS Secret it could not load, an annotation it rejected, a rule it dropped. When an Ingress applies cleanly and behaves as though it does not exist, that is where the reason is.

    bash Example session 
    kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=10 2>&1 | grep -oE '"GET [^"]*" [0-9]{3}' | tail -5"GET /green/ HTTP/1.1" 200"GET /blue/ HTTP/1.1" 200"GET /nothing/ HTTP/1.1" 404"GET /green/ HTTP/1.1" 503"GET /blue/ HTTP/1.1" 200

    Expected resultThe session's requests in order with their status codes. Note the 404 for /nothing/ sits in the same log as the 503 for /green/, which is how you compare a routing miss against a backend miss without running anything.

    Success conditionYou can read the sequence of requests and statuses out of the controller's log.

  5. Put the checks in one order

    Working from the status code, the sequence is short:

    On a 404, first establish who wrote it. Read the body for a backend fingerprint, and ask the backend directly through its Service from a Pod in the namespace. A backend 404 is an application path problem, usually needing rewrite-target or a base-path setting. Only if the 404 is the controller's do you check the rule: is the Host header what you think, is pathType what you think, and does the Ingress even have a controller. An Ingress whose ingressClassName names no installed controller has an empty ADDRESS and returns 404 for everything, which the Ingress controllers guide covers.

    On a 503, go to the endpoints. kubectl get endpointslice -n -l kubernetes.io/service-name=. Empty means readiness, selector, or scheduling. Populated means look at the port numbers in the Ingress and the Service, which is the one 503 the endpoints check does not explain.

    On a 502, the connection happened. Check the Pod's actual listening port against the Service's targetPort, then the protocol: an HTTPS backend behind a controller speaking HTTP produces a 502, and needs nginx.ingress.kubernetes.io/backend-protocol: HTTPS. Then the Pod's own logs, because an application crashing mid-request is a 502 too.

    On a 504, the backend is slow. Confirm from the upstream response time in the access log before raising proxy-read-timeout, so you are treating a timeout rather than hiding a performance problem.

    When there is no code at all, curl reporting 000 or hanging, nothing completed. Look in front of the controller: DNS, the node port, the load balancer, NetworkPolicy. The access log settles it, because a request that never appears never arrived.

    One habit is worth more than the list. Test each hop separately. Client to controller, controller to Service, Service to Pod. Curl from a Pod in the namespace to http:/// to skip the Ingress; curl to a Pod IP directly to skip the Service. Whichever hop first misbehaves is where the bug is, and that narrowing is faster than reasoning about the whole path at once.

    bash Example session 
    kubectl scale deploy green -n ing --replicas=1; kubectl rollout status deploy/green -n ing --timeout=180s | tail -1deployment.apps/green scaleddeployment "green" successfully rolled out

    Expected resultThe backend restored, and the 503 gone with it. Scaling back up is the whole fix when the cause was zero replicas.

    Success conditionEndpoints return and the path serves 200 again.

Troubleshooting

Official sources

← All CKA guides