Ingress Path Routing and Rewrites
An Ingress object is a routing table that does nothing on its own - a controller has to be running to read it. Once one is, a /blue path returns 404 from a perfectly healthy backend, and the fix is one annotation.
Networking and Discovery Guide 20 of 46 Intermediate
- Kubernetes1.36.4
- Cluster4 nodes
- Runtimecontainerd 2.2.6
- CNICalico v3.32.1
- Controlleringress-nginx v1.13.1
- TimeAbout 18 min
- Reviewed23 August 2026
Written against the versions above. The Ingress API is stable and identical everywhere. The `rewrite-target` annotation is NOT - it is nginx-specific, and every controller spells path rewriting differently. That inconsistency is the main reason the Gateway API exists.
| Server Name | IP Address | OS | Roles | CPU | RAM | HDD |
|---|---|---|---|---|---|---|
| CKA1001 | 192.168.0.175 | Ubuntu 26.04 LTS | Control Plane Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE01 | 192.168.0.176 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE02 | 192.168.0.177 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
| CKA1001-NODE03 | 192.168.0.178 | Ubuntu 26.04 LTS | Worker Node | 2 Core | 4 GB | 50 GB |
Before you start
- An Ingress controller installed. Nothing about an Ingress object works without one, which is the first step of this guide.
- Two Services to route to. This session uses an
ingnamespace withblueandgreenDeployments, each serving a page that names itself, exposed as ClusterIP Services on port 80.
-
The controller is just a Pod, and IngressClass names it
Before any routing, the thing that does the routing:
ingress-nginx-controller-7df97f6c86-bz2n5 RunningAn Ingress controller is an ordinary Deployment. It watches the API for Ingress objects and reconfigures itself. Kubernetes ships the *Ingress API* and no implementation, so a cluster with no controller accepts every Ingress you write and routes nothing.
It is reached through its own Service - here a NodePort, so 30561 is the way in:
NodePort 30561 31967And an IngressClass is how an Ingress says which controller should pick it up:
nginx k8s.io/ingress-nginxThe left column is the name an Ingress puts in
spec.ingressClassName; the right is the controller that claims it. Clusters run more than one controller often enough - an internal and an external one - that leaving the class off is a real way to have an Ingress silently claimed by nobody.bash Example session kubectl get pods -n ingress-nginx --no-headers | awk '{print $1, $3}'ingress-nginx-controller-7df97f6c86-bz2n5 Runningkubectl get svc ingress-nginx-controller -n ingress-nginx -o custom-columns=TYPE:.spec.type,HTTP:.spec.ports[0].nodePort,HTTPS:.spec.ports[1].nodePort --no-headersNodePort 30561 31967kubectl get ingressclass -o custom-columns=NAME:.metadata.name,CONTROLLER:.spec.controller --no-headersnginx k8s.io/ingress-nginxExpected resultA running controller Pod, its NodePorts, and an IngressClass naming the controller.
Success conditionYou can say what happens to an Ingress in a cluster with no controller.
-
Two paths, and a 404 that is not the controller's
The Ingress routes
/blueto the blue Service and/greento green, on hostshop.cg.test. It gets an address, so the controller has claimed it:NAME CLASS HOSTS ADDRESS PORTS AGE shop nginx shop.cg.test 192.168.0.178 80 15sAnd the request fails:
/blue/ -> 404Here is the question worth asking, and it is the one people skip: whose 404 is this? Read the server banner:
<hr><center>nginx/1.29.8</center>That is a version. The controller's own error pages do not carry one. This 404 came from the backend - your application - which means routing worked perfectly and the backend was asked for something it does not have.
Prove it from inside the cluster, bypassing the Ingress entirely. The Service root works, and the path the Ingress forwards does not:
kubectl run t1 ... wget -qO- http://blue/ # serves the page kubectl run t2 ... wget -qO- http://blue/blue/ # 404By default an Ingress passes the whole path through. A request for
/blue/arrives at the backend as/blue/, and the backend has nothing there.bash Example session sleep 15; kubectl get ingress -n ingNAME CLASS HOSTS ADDRESS PORTS AGEshop nginx shop.cg.test 192.168.0.178 80 15scurl -s -o /dev/null -w '/blue/ -> %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue//blue/ -> 404curl -s --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/ | grep -iE 'title|center'<head><title>404 Not Found</title></head><center><h1>404 Not Found</h1></center><hr><center>nginx/1.29.8</center>kubectl run t1 -n ing --rm -i --restart=Never --image=busybox:1.36 -- wget -qO- --timeout=5 http://blue/ 2>&1 | head -2<!DOCTYPE html><html>kubectl run t2 -n ing --rm -i --restart=Never --image=busybox:1.36 -- wget -qO- --timeout=5 http://blue/blue/ 2>&1 | tail -3wget: server returned error: HTTP/1.1 404 Not Foundpod "t2" deleted from ing namespacepod ing/t2 terminated (Error)Expected resultA 404 carrying a version banner, and the same 404 reproducible against the Service directly.
Success conditionYou attributed the 404 to the backend without guessing.
-
One annotation, and the difference between two 404s
The fix is to rewrite the path before it reaches the backend -
nginx.ingress.kubernetes.io/rewrite-target, with a capture group in the path. Apply it and the same request works:right host: 200Now the two failure cases, which look identical from a browser and are not.
A path the Ingress does not mention:
unmatched path: 404A host the Ingress does not mention:
<hr><center>nginx</center>Note the banner:
nginxwith no version. That is the controller's own default backend saying "no rule matched", not an application. So the version string is a genuinely useful diagnostic - a 404 with a version came from your workload and the routing is fine; a 404 without one means nothing matched and the request never left the controller.Worth remembering:
rewrite-targetis an nginx annotation. Nothing about it is portable, and a different controller will ignore it entirely while appearing to accept the Ingress.bash Example session kubectl apply -f /tmp/ing-fixed.yamlingress.networking.k8s.io/shop configuredcurl -s -o /dev/null -w 'right host: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/right host: 200curl -s -o /dev/null -w 'unmatched path: %{http_code}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/nothing/unmatched path: 404curl -s --max-time 8 -H 'Host: nosuchhost.cg.test' http://127.0.0.1:30561/ | grep -iE 'center|title|404'<head><title>404 Not Found</title></head><center><h1>404 Not Found</h1></center><hr><center>nginx</center>Expected resultA 200 after the rewrite, and a versionless 404 banner for an unknown host.
Success conditionYou can tell the controller's 404 from the backend's by reading the banner.
-
TLS, and a redirect you did not ask for
An Ingress terminates TLS by naming a Secret. The Secret has a required type and two required keys:
kubernetes.io/tls tls.crt tls.keyWith
spec.tlspointing at it, HTTPS serves the app:I am BLUEAnd now the surprise. The plain HTTP request that returned 200 in the last step has changed behaviour:
http now: 308 -> https://shop.cg.test/blueAdding TLS silently turned on an HTTP-to-HTTPS redirect. Nobody configured that; ingress-nginx does it by default the moment a host has a certificate. It is usually what you want and it is occasionally the cause of a redirect loop behind another proxy that is also terminating TLS - the annotation to turn it off is
ssl-redirect: "false".One more, because it explains a class of confusion: this certificate is self-signed, so a client that checks properly refuses it.
without -k: 000000is not an HTTP status - it means no HTTP response happened at all. The TLS handshake failed, so there was never a request to answer. A000from curl is always a connection-level problem rather than a server one.bash Example session kubectl create secret tls shop-tls -n ing --cert=/tmp/tls.crt --key=/tmp/tls.keysecret/shop-tls createdkubectl get secret shop-tls -n ing -o go-template='{{.type}}{{"\n"}}{{range $k,$v := .data}}{{$k}} {{end}}{{"\n"}}'kubernetes.io/tlstls.crt tls.key curl -sk --max-time 8 --resolve shop.cg.test:31967:127.0.0.1 https://shop.cg.test:31967/blue/I am BLUEcurl -s -o /dev/null -w 'http now: %{http_code} -> %{redirect_url}\n' --max-time 8 -H 'Host: shop.cg.test' http://127.0.0.1:30561/blue/http now: 308 -> https://shop.cg.test/bluecurl -s -o /dev/null -w 'without -k: %{http_code}\n' --max-time 8 --resolve shop.cg.test:31967:127.0.0.1 https://shop.cg.test:31967/blue/ 2>&1 | tail -2without -k: 000Expected resultHTTPS serving the app, HTTP answering 308, and a rejected self-signed certificate returning 000.
Success conditionYou can explain the 308 nobody configured.
Troubleshooting
An Ingress exists, has no ADDRESS, and nothing routes.
Why: No controller has claimed it - either none is installed, or its
ingressClassNamenames a class no controller implements.Fix:
kubectl get ingressclassfor what exists, and compare withkubectl get ingress <name> -o jsonpath='{.spec.ingressClassName}'. An empty ADDRESS column means nobody is listening.A path returns 404 and the backend is healthy.
Why: The full path is being passed through. The backend is asked for /blue/ and only serves /.
Fix:Read the 404's server banner: a version means it came from your workload. Add a path rewrite - on ingress-nginx,
nginx.ingress.kubernetes.io/rewrite-targetwith a capture group.Requests started redirecting to HTTPS and nobody configured that.
Why: ingress-nginx enables an HTTP-to-HTTPS redirect automatically once a host has a TLS certificate.
Fix:Leave it on unless something in front of the cluster is already terminating TLS - two redirects in a chain is how you get a loop. To disable it, annotate the Ingress with
nginx.ingress.kubernetes.io/ssl-redirect: "false".curl returns 000 rather than a status code.
Why: No HTTP response happened - a TLS handshake failure, a refused connection or a timeout. With a self-signed certificate it is the handshake.
Fix:Retry with
-kto confirm it is certificate validation, then fix the trust chain rather than leaving -k in place.