CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Security Specialist

Protecting Node Metadata and Endpoints

Every Pod can address the node it runs on, and on a cloud cluster it can address the metadata service that hands out credentials. This guide probes three node endpoints from inside a Pod, reads the kubelet settings that decide the answers, and then closes the path with a single egress rule.

Cluster Setup Guide 7 of 40 Advanced

Written against the versions above. These are bare-metal clusters, so there is no cloud metadata service at 169.254.169.254 - the guide shows that measurement honestly and applies the control that would protect it where one exists.

The cka6001 cluster: one control plane and 2 schedulable workers, on Cilium.
Server NameIP AddressOSRolesCPURAMHDD
CKA6001192.168.0.46Ubuntu 26.04 LTSControl Plane Node (tainted NoSchedule)2 Core4 GB50 GB
CKA6001-NODE01192.168.0.47Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA6001-NODE02192.168.0.48Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. Three endpoints, probed from inside a Pod

    The node's own address, from a workload running on it.

    bash Example session 
    kubectl create ns cks-node >/dev/null 2>&1; kubectl -n cks-node run probe --image=docker.io/curlimages/curl:8.11.0 --restart=Never --command -- sh -c 'sleep 900' >/dev/null; kubectl -n cks-node run peer --image=docker.io/library/busybox:1.37 --labels=app=peer --restart=Never --command -- sh -c 'echo PEER-OK > /tmp/index.html; httpd -f -h /tmp -p 8080' >/dev/null; kubectl -n cks-node wait --for=condition=Ready pod/probe pod/peer --timeout=150s >/dev/null; kubectl get node cka6001-node01 -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'; echo " <- the node a Pod can address directly"192.168.0.47 <- the node a Pod can address directlyN=$(kubectl get node cka6001-node01 -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'); echo -n "kubelet API  10250: "; kubectl -n cks-node exec probe -- curl -sk --max-time 6 -o /dev/null -w '%{http_code}\n' https://$N:10250/pods; echo -n "read-only    10255: "; kubectl -n cks-node exec probe -- sh -c "curl -s --max-time 5 -o /dev/null -w '%{http_code}' http://$N:10255/pods 2>&1 || echo ' connection refused'"; echo -n "metadata 169.254.169.254: "; kubectl -n cks-node exec probe -- sh -c "curl -s --max-time 5 -o /dev/null -w '%{http_code}' http://169.254.169.254/ 2>&1 || echo ' no route, timed out'"; echo "--- 401 means the endpoint is THERE and asking for credentials. The other two are not answering at all"kubelet API  10250: 401read-only    10255: 000 connection refusedmetadata 169.254.169.254: 000 no route, timed out--- 401 means the endpoint is THERE and asking for credentials. The other two are not answering at all

    Expected resultkubelet API 10250: 401, read-only 10255: 000 connection refused, and metadata 169.254.169.254: 000 no route, timed out.

    Success conditionYou know which node endpoints answer a Pod, and how.

  2. Why the kubelet answered 401 rather than 200

    The settings behind that status code.

    bash Example session 
    sudo grep -A6 '^authentication:' /var/lib/kubelet/config.yaml | head -8; sudo grep -A2 '^authorization:' /var/lib/kubelet/config.yaml | head -3; echo "--- anonymous OFF, x509 client certs, and authorization delegated to the apiserver by webhook. That combination is what turns 10250 from a data leak into a 401"authentication:  anonymous:    enabled: false  webhook:    cacheTTL: 0s    enabled: true  x509:authorization:  mode: Webhook  webhook:--- anonymous OFF, x509 client certs, and authorization delegated to the apiserver by webhook. That combination is what turns 10250 from a data leak into a 401echo -n "readOnlyPort settings in the kubelet config: "; sudo grep -c readOnlyPort /var/lib/kubelet/config.yaml; echo "--- zero, so the unauthenticated read-only port is off. When it is on it serves the full Pod list to anyone who can reach the node"readOnlyPort settings in the kubelet config: 0--- zero, so the unauthenticated read-only port is off. When it is on it serves the full Pod list to anyone who can reach the node

    Expected resultanonymous: enabled: false, webhook: enabled: true, an x509: block and authorization: mode: Webhook. Then 0 occurrences of readOnlyPort.

    Success conditionYou can explain a kubelet's exposure from its config file.

  3. Allow the cluster, deny the infrastructure

    One egress policy with an exception list.

    bash Example session 
    printf 'apiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: deny-node-and-metadata\n  namespace: cks-node\nspec:\n  podSelector: {}\n  policyTypes: ["Egress"]\n  egress:\n  - to:\n    - ipBlock:\n        cidr: 0.0.0.0/0\n        except:\n        - 169.254.169.254/32\n        - 192.168.0.0/24\n  - to:\n    - namespaceSelector: {}\n' | kubectl apply -f - 2>&1 | tail -1; sleep 10; echo "--- everything is allowed EXCEPT the link-local metadata address and the node network. The second rule keeps pod-to-pod and DNS working"networkpolicy.networking.k8s.io/deny-node-and-metadata created--- everything is allowed EXCEPT the link-local metadata address and the node network. The second rule keeps pod-to-pod and DNS workingN=$(kubectl get node cka6001-node01 -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'); echo -n "kubelet API  10250: "; kubectl -n cks-node exec probe -- sh -c "curl -sk --max-time 6 -o /dev/null -w '%{http_code}' https://$N:10250/pods 2>&1 || echo ' timed out'"; P=$(kubectl -n cks-node get pod peer -o jsonpath='{.status.podIP}'); echo -n "peer Pod     8080:  "; kubectl -n cks-node exec probe -- sh -c "curl -s --max-time 6 http://$P:8080/ 2>&1 | tail -1"; echo "--- the node endpoint that answered 401 a moment ago now answers nothing, and ordinary Pod traffic is untouched"kubelet API  10250: 000 timed outpeer Pod     8080:  PEER-OK--- the node endpoint that answered 401 a moment ago now answers nothing, and ordinary Pod traffic is untouched

    Expected resultAfter the policy, kubelet API 10250: 000 timed out - and peer Pod 8080: PEER-OK.

    Success conditionThe node layer is closed and the workload still works.

Troubleshooting

Official sources

← All CKS guides