CertGrid CertGrid
Hands-on Lab·Kubernetes and Cloud Native Associate

Ingress Path Routing and Rewrites

An Ingress object is a routing table that does nothing on its own - a controller has to be running to read it. Once one is, a /blue path returns 404 from a perfectly healthy backend, and the fix is one annotation.

Networking and Discovery Guide 20 of 46 Intermediate

Written against the versions above. The Ingress API is stable and identical everywhere. The `rewrite-target` annotation is NOT - it is nginx-specific, and every controller spells path rewriting differently. That inconsistency is the main reason the Gateway API exists.

The controller runs as a Pod on this cluster and is reached through its own NodePort, 30561 for HTTP and 31967 for HTTPS.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. The controller is just a Pod, and IngressClass names it

    Before any routing, the thing that does the routing:

    ingress-nginx-controller-7df97f6c86-bz2n5 Running

    An Ingress controller is an ordinary Deployment. It watches the API for Ingress objects and reconfigures itself. Kubernetes ships the *Ingress API* and no implementation, so a cluster with no controller accepts every Ingress you write and routes nothing.

    It is reached through its own Service - here a NodePort, so 30561 is the way in:

    NodePort   30561   31967

    And an IngressClass is how an Ingress says which controller should pick it up:

    nginx   k8s.io/ingress-nginx

    The left column is the name an Ingress puts in spec.ingressClassName; the right is the controller that claims it. Clusters run more than one controller often enough - an internal and an external one - that leaving the class off is a real way to have an Ingress silently claimed by nobody.

    bash Example session
    kubectl get pods -n ingress-nginx --no-headers | awk '{print $1, $3}'ingress-nginx-controller-7df97f6c86-bz2n5 Runningkubectl get svc ingress-nginx-controller -n ingress-nginx -o custom-columns=TYPE:.spec.type,HTTP:.spec.ports[0].nodePort,HTTPS:.spec.ports[1].nodePort --no-headersNodePort   30561   31967kubectl get ingressclass -o custom-columns=NAME:.metadata.name,CONTROLLER:.spec.controller --no-headersnginx   k8s.io/ingress-nginx

    Expected resultA running controller Pod, its NodePorts, and an IngressClass naming the controller.

    Success conditionYou can say what happens to an Ingress in a cluster with no controller.

  2. Two paths, and a 404 that is not the controller's

    The Ingress routes /blue to the blue Service and /green to green, on host shop.cg.test. It gets an address, so the controller has claimed it:

    NAME   CLASS   HOSTS          ADDRESS         PORTS   AGE
    shop   nginx   shop.cg.test   192.168.0.178   80      15s

    And the request fails:

    /blue/ -> 404

    Here is the question worth asking, and it is the one people skip: whose 404 is this? Read the server banner:

    <hr><center>nginx/1.29.8</center>

    That is a version. The controller's own error pages do not carry one. This 404 came from the backend - your application - which means routing worked perfectly and the backend was asked for something it does not have.

    Prove it from inside the cluster, bypassing the Ingress entirely. The Service root works, and the path the Ingress forwards does not:

    kubectl run t1 ... wget -qO- http://blue/        # serves the page
    kubectl run t2 ... wget -qO- http://blue/blue/  # 404

    By default an Ingress passes the whole path through. A request for /blue/ arrives at the backend as /blue/, and the backend has nothing there.

    bash Example session
    sleep 15; kubectl get ingress -n ingNAME   CLASS   HOSTS          ADDRESS         PORTS   AGEshop   nginx   shop.cg.test   192.168.0.178   80      15scurl -s -o /dev/null -w '/blue/ -> %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue//blue/ -> 404curl -s --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/ | grep -iE 'title|center'<head><title>404 Not Found</title></head><center><h1>404 Not Found</h1></center><hr><center>nginx/1.29.8</center>kubectl run t1 -n ing --rm -i --restart=Never --image=busybox:1.36 -- wget -qO- --timeout=5 http://blue/ 2>&1 | head -2<!DOCTYPE html><html>kubectl run t2 -n ing --rm -i --restart=Never --image=busybox:1.36 -- wget -qO- --timeout=5 http://blue/blue/ 2>&1 | tail -3wget: server returned error: HTTP/1.1 404 Not Foundpod "t2" deleted from ing namespacepod ing/t2 terminated (Error)

    Expected resultA 404 carrying a version banner, and the same 404 reproducible against the Service directly.

    Success conditionYou attributed the 404 to the backend without guessing.

  3. One annotation, and the difference between two 404s

    The fix is to rewrite the path before it reaches the backend - nginx.ingress.kubernetes.io/rewrite-target, with a capture group in the path. Apply it and the same request works:

    right host: 200

    Now the two failure cases, which look identical from a browser and are not.

    A path the Ingress does not mention:

    unmatched path: 404

    A host the Ingress does not mention:

    <hr><center>nginx</center>

    Note the banner: nginx with no version. That is the controller's own default backend saying "no rule matched", not an application. So the version string is a genuinely useful diagnostic - a 404 with a version came from your workload and the routing is fine; a 404 without one means nothing matched and the request never left the controller.

    Worth remembering: rewrite-target is an nginx annotation. Nothing about it is portable, and a different controller will ignore it entirely while appearing to accept the Ingress.

    bash Example session
    kubectl apply -f /tmp/ing-fixed.yamlingress.networking.k8s.io/shop configuredcurl -s -o /dev/null -w 'right host: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/right host: 200curl -s -o /dev/null -w 'unmatched path: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/nothing/unmatched path: 404curl -s --max-time 8 -H 'Host: nosuchhost.cg.test' http://127.0.0.1:30561/ | grep -iE 'center|title|404'<head><title>404 Not Found</title></head><center><h1>404 Not Found</h1></center><hr><center>nginx</center>

    Expected resultA 200 after the rewrite, and a versionless 404 banner for an unknown host.

    Success conditionYou can tell the controller's 404 from the backend's by reading the banner.

  4. TLS, and a redirect you did not ask for

    An Ingress terminates TLS by naming a Secret. The Secret has a required type and two required keys:

    kubernetes.io/tls
    tls.crt tls.key

    With spec.tls pointing at it, HTTPS serves the app:

    I am BLUE

    And now the surprise. The plain HTTP request that returned 200 in the last step has changed behaviour:

    http now: 308 -> https://shop.cg.test/blue

    Adding TLS silently turned on an HTTP-to-HTTPS redirect. Nobody configured that; ingress-nginx does it by default the moment a host has a certificate. It is usually what you want and it is occasionally the cause of a redirect loop behind another proxy that is also terminating TLS - the annotation to turn it off is ssl-redirect: "false".

    One more, because it explains a class of confusion: this certificate is self-signed, so a client that checks properly refuses it.

    without -k: 000

    000 is not an HTTP status - it means no HTTP response happened at all. The TLS handshake failed, so there was never a request to answer. A 000 from curl is always a connection-level problem rather than a server one.

    bash Example session
    kubectl create secret tls shop-tls -n ing --cert=/tmp/tls.crt --key=/tmp/tls.keysecret/shop-tls createdkubectl get secret shop-tls -n ing -o go-template='{{.type}}{{"\n"}}{{range $k,$v := .data}}{{$k}} {{end}}{{"\n"}}'kubernetes.io/tlstls.crt tls.key curl -sk --max-time 8 --resolve shop.cg.test:31967:127.0.0.1 https://shop.cg.test:31967/blue/I am BLUEcurl -s -o /dev/null -w 'http now: %{http_code} -> %{redirect_url}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/http now: 308 -> https://shop.cg.test/bluecurl -s -o /dev/null -w 'without -k: %{http_code}\n' --max-time 8 --resolve shop.cg.test:31967:127.0.0.1 https://shop.cg.test:31967/blue/ 2>&1 | tail -2without -k: 000

    Expected resultHTTPS serving the app, HTTP answering 308, and a rejected self-signed certificate returning 000.

    Success conditionYou can explain the 308 nobody configured.

Troubleshooting

Official sources