CertGrid CertGrid
Troubleshooting·Certified Kubernetes Administrator

Troubleshooting NetworkPolicy-Blocked Traffic

A NetworkPolicy denial times out rather than refusing, which makes it look like a dead backend. An egress policy that blocks DNS fails differently again. Those two signatures, plus knowing policies only ever add permission, cover most of the diagnosis.

Troubleshooting Guide 99 of 103 Intermediate

Written against the versions above. Enforcement is the CNI's job, so the API is portable and the exact failure timing is not. Calico is the enforcer here.

Two namespaces on the four-node cluster: `npt` holding the server and the policies, `npt-other` for the cross-namespace test.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. The baseline, and the first command to run

    With no policy selecting them, Pods can reach anything:

    server reached

    And the count of policies in the whole cluster:

    1

    That one policy is in the default namespace, left over from a different exercise, and it is worth pausing on because it demonstrates the property that catches people out. NetworkPolicy is namespaced. A policy in default has no effect on Pods in npt, so the count being non-zero says nothing about whether npt is affected.

    That makes kubectl get netpol -A the first command when connectivity is unexplained. It takes a second and it answers a question the rest of the debugging depends on:

    • No policies anywhere and the problem is not NetworkPolicy. Go to Services, DNS or the application.
    • Policies exist, and one selects the Pod in question, and you have a candidate.

    The -A matters twice over: for the traffic you care about, both the source namespace and the destination namespace can hold a policy that blocks it, and the two are enforced independently. A policy on the client's egress and a policy on the server's ingress are separate decisions, and both must permit the connection.

    One prerequisite that is easy to forget. The NetworkPolicy API accepts objects whether or not anything enforces them. On a cluster with a CNI that does not implement policy, every policy you write applies cleanly and nothing changes. If policies exist and appear to do nothing at all, check the CNI before checking the policies.

    bash Example session 
    kubectl get netpol -A --no-headers 2>/dev/null | wc -l1kubectl get netpol -n default --no-headers 2>&1default-deny   <none>   6h31mkubectl logs base -n nptserver reached

    Expected resultOne policy in the cluster, in a namespace unrelated to the test, and an unrestricted Pod reaching the server. The probe Pod ran wget -qO- --timeout=5 http://server/.

    Success conditionYou can count the policies in the cluster and confirm connectivity works before any are added.

  2. The signature: it times out, it does not refuse

    Apply a default-deny for ingress and repeat the same request:

    wget: download timed out
Command exited with non-zero status 1
real    0m 5.00s

    Timed out, and the real 0m 5.00s is exactly the --timeout=5 that was asked for, meaning the client waited the full budget and gave up.

    This is the single most useful thing to know about NetworkPolicy failures, because the alternative is very different. A refused connection means something answered: a TCP RST came back, so the packet reached a host that had no listener. A timeout means nothing answered at all: packets went out and were dropped in silence, and the client waited.

    So the two failures point in opposite directions:

    • Connection refused - routing works, nothing is listening on that port. Wrong port, application not started, wrong targetPort.
    • Timeout - packets are being dropped. A NetworkPolicy, a firewall, a broken route, or a Service with no endpoints backing an address that is nonetheless routable.

    That is why NetworkPolicy denials get misdiagnosed as dead backends. The symptom is identical to a hung application, and people go to read application logs, where there is nothing at all, because the request never arrived.

    One detail to be careful with. A policy drops packets, it does not close connections. Established connections often continue working after a policy is applied, because conntrack already has an entry for them. So a policy can break every new connection while long-lived ones look healthy, which makes the change look harmless right after it is applied and broken later when something reconnects. When testing a policy, always test with a fresh connection.

    And the deny itself is four lines with nothing in it:

    spec:
  podSelector: {}
  policyTypes: [Ingress]

    podSelector: {} is an empty selector, which matches every Pod in the namespace, not none. policyTypes: [Ingress] with no ingress rules is the deny: it isolates the selected Pods for ingress and then permits nothing. Written by hand this is a two-character difference from something that has no effect, and it is a common accident.

    bash Example session 
    kubectl apply -f - <<'EOF'apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: default-deny-ingress  namespace: nptspec:  podSelector: {}  policyTypes: [Ingress]EOFnetworkpolicy.networking.k8s.io/default-deny-ingress createdkubectl logs denied -n npt 2>&1 | head -4wget: download timed outCommand exited with non-zero status 1real	0m 5.00suser	0m 0.00s

    Expected resultA timeout after exactly the client's timeout budget, not a refusal. The probe ran time wget -qO- --timeout=5 http://server/, so the real line is the request's own duration.

    Success conditionYou can produce a policy denial and recognise it as a timeout rather than a refusal.

  3. describe explains the policy in words

    Two commands identify the culprit. The listing, showing selector and types:

    default   default-deny           map[]   [Ingress]
npt       default-deny-ingress   map[]   [Ingress]

    map[] is how an empty podSelector prints, and it means all Pods in that namespace. Both policies here are namespace-wide denies, and reading this table is usually enough to know whether a policy could be responsible.

    Then describe, which is unusually good for this resource because it translates the semantics into sentences:

    Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing ingress traffic:
    <none> (Selected pods are isolated for ingress connectivity)
  Not affecting egress traffic
  Policy Types: Ingress

    Three lines worth reading closely.

    PodSelector: (... to all pods in this namespace) confirms the empty selector means all, which is the opposite of what "none" suggests on its own.

    Allowing ingress traffic: (Selected pods are isolated for ingress connectivity) is the deny stated plainly. "Isolated" is the term the API uses: once any policy selects a Pod for a direction, that Pod is isolated in that direction, and only explicit rules let traffic back in.

    Not affecting egress traffic is the reassurance that this policy does not touch outbound. Reading this on every candidate policy is how you build the picture of which directions are actually restricted, and the phrasing makes it hard to misread.

    On a cluster with many policies, kubectl describe netpol -n describes all of them at once, which is faster than reading YAML and much harder to misinterpret.

    bash Example session 
    kubectl get netpol -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,POD-SELECTOR:.spec.podSelector,TYPES:.spec.policyTypes --no-headersdefault   default-deny           map[]   [Ingress]npt       default-deny-ingress   map[]   [Ingress]kubectl describe netpol default-deny-ingress -n npt | head -14Name:         default-deny-ingressNamespace:    nptCreated on:   2026-08-21 12:07:08 +0000 UTCLabels:       <none>Annotations:  <none>Spec:  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)  Allowing ingress traffic:    <none> (Selected pods are isolated for ingress connectivity)  Not affecting egress traffic  Policy Types: Ingress

    Expected resultThe selector printed as map[] in the listing and explained as "all pods in this namespace" by describe, with the deny stated as isolation.

    Success conditionYou can identify which policies select a Pod and read what each one permits.

  4. Policies only ever add permission

    The instinct on finding a deny policy is to edit it. That is the wrong move, and understanding why prevents a class of mistakes.

    There is no such thing as a deny rule in NetworkPolicy. Every rule grants. The effective permission for a Pod is the union of every rule in every policy that selects it, and a Pod becomes isolated the moment any policy selects it for a direction. So a deny-all is simply a policy that selects Pods and grants nothing, and the fix is to add another policy that grants what is needed:

    spec:
  podSelector: {matchLabels: {app: server}}
  policyTypes: [Ingress]
  ingress:
  - from:
    - podSelector: {matchLabels: {role: client}}
    ports:
    - {protocol: TCP, port: 80}

    Both policies now select the server Pod, and the result is the union:

    with role=client: server reached
without the label: FAILED

    Two policies in the namespace, neither modified:

    allow-from-client
default-deny-ingress

    This has three consequences worth internalising.

    You cannot make an exception. "Allow everything except this one Pod" is not expressible in a NetworkPolicy. Deny-by-default plus explicit allows is the only shape. Calico's own CRDs add ordered policies with deny actions, and so do some other CNIs, but that is beyond the portable API.

    Adding a policy can only widen access, never narrow it. So a new policy cannot be the cause of something that stopped working, unless it is the *first* policy to select those Pods, which flips them from unisolated to isolated. That single exception is where nearly all NetworkPolicy incidents come from.

    Removing an allow policy narrows access, which is the reverse of the intuition people carry from firewalls. Deleting what looks like an unused policy is how working traffic gets broken.

    The practical debugging move follows from the union rule: to find out why a Pod is unreachable, list every policy selecting it and check whether any of them permits the traffic. One permitting rule is enough; the others are irrelevant.

    And note what the probes prove about matching. Both probe Pods were identical busybox Pods in the same namespace, and the only difference was the role=client label. The podSelector in a from block matches on labels and nothing else: not names, not IPs, not ServiceAccounts. A Pod that should have access and does not, when the policy looks right, is nearly always a label that does not match, and kubectl get pod --show-labels settles it in one command.

    bash Example session 
    kubectl apply -f - <<'EOF'apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: allow-from-client  namespace: nptspec:  podSelector: {matchLabels: {app: server}}  policyTypes: [Ingress]  ingress:  - from:    - podSelector: {matchLabels: {role: client}}    ports:    - {protocol: TCP, port: 80}EOFnetworkpolicy.networking.k8s.io/allow-from-client createdecho "with role=client: $(kubectl logs allowed -n npt 2>&1 | tail -1)"; echo "without the label: $(kubectl logs unlabelled -n npt 2>&1 | tail -1)"with role=client: server reachedwithout the label: FAILEDkubectl get netpol -n npt --no-headers | awk '{print $1}'allow-from-clientdefault-deny-ingress

    Expected resultThe labelled Pod through, the unlabelled Pod still blocked, and the deny policy untouched. Both Pods were the same image in the same namespace.

    Success conditionAdding an allow policy restores access for matching Pods without editing the deny.

  5. The egress trap: DNS fails, and it fails differently

    Now the most common way a NetworkPolicy rollout breaks things, and the reason to read the failure text rather than just noting that it failed.

    Apply a default-deny for egress on the client and try again:

    wget: bad address 'server'

    Not a timeout. bad address is a name resolution failure, and it happened before any connection to the server was attempted.

    The reason is that DNS is egress. A Pod resolving server sends a UDP packet to the CoreDNS Service on port 53, and an egress policy that grants nothing blocks that too. So a policy meant to restrict where an application can connect breaks its ability to look up anything at all, including the destinations you intended to allow.

    The two signatures are worth committing to memory, because they split the problem in half instantly:

    • bad address, Name or service not known, Temporary failure in name resolution - DNS. The egress policy is missing a rule for port 53 to kube-dns.
    • Timed out on a connection to an address that resolved - the policy is blocking the application traffic itself, or the destination's ingress policy is.

    The fix needs both rules, and note the to selector for DNS:

      egress:
  - to:
    - namespaceSelector:
        matchLabels: {kubernetes.io/metadata.name: kube-system}
    ports:
    - {protocol: UDP, port: 53}
    - {protocol: TCP, port: 53}
  - to:
    - podSelector: {matchLabels: {app: server}}
    ports:
    - {protocol: TCP, port: 80}

    With both, resolution and the request work:

    Server:     10.96.0.10
Address:    10.96.0.10:53
Name:   server.npt.svc.cluster.local
Address: 10.104.91.161
server reached

    Four details in that DNS rule that each cause their own failure.

    kubernetes.io/metadata.name is a label Kubernetes sets automatically on every namespace, holding the namespace's own name. It is the reliable way to select a namespace, because it is always there. Selecting on a hand-applied label like name=kube-system works only if somebody applied it, and a namespace with no labels cannot be selected at all.

    Both UDP and TCP on 53. DNS uses UDP for ordinary queries and falls back to TCP for large responses. Allowing only UDP works until a response exceeds the UDP limit, and then fails intermittently in a way that is very hard to attribute.

    A namespaceSelector, not a podSelector. CoreDNS runs in kube-system, and a bare podSelector in an egress rule only ever matches Pods in the policy's own namespace. This is the same limitation step 6 covers, and getting it wrong here is why so many DNS allow-rules silently do nothing.

    The Service ClusterIP is not what the policy matches. Pods talk to 10.96.0.10, the kube-dns Service, but policy is enforced on the Pod IPs behind it, which is why selecting the CoreDNS Pods by namespace is the right approach and an ipBlock for the Service CIDR generally is not.

    Almost every cluster running policies has a namespace-wide allow-DNS-egress policy for exactly this reason. It is close to boilerplate, and writing it once per namespace is cheaper than diagnosing it once.

    bash Example session 
    kubectl apply -f - <<'EOF'apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: deny-egress-from-client  namespace: nptspec:  podSelector: {matchLabels: {role: client}}  policyTypes: [Egress]EOFnetworkpolicy.networking.k8s.io/deny-egress-from-client createdkubectl logs noegress -n npt 2>&1 | head -5wget: bad address 'server'kubectl apply -f - <<'EOF'apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: allow-dns-and-server  namespace: nptspec:  podSelector: {matchLabels: {role: client}}  policyTypes: [Egress]  egress:  - to:    - namespaceSelector:        matchLabels: {kubernetes.io/metadata.name: kube-system}    ports:    - {protocol: UDP, port: 53}    - {protocol: TCP, port: 53}  - to:    - podSelector: {matchLabels: {app: server}}    ports:    - {protocol: TCP, port: 80}EOFnetworkpolicy.networking.k8s.io/allow-dns-and-server createdkubectl logs fixed2 -n npt 2>&1 | grep -vE '^$' | tail -6Server:		10.96.0.10Address:	10.96.0.10:53Name:	server.npt.svc.cluster.localAddress: 10.104.91.161server reached

    Expected resultbad address under the egress deny, then a successful lookup and request once DNS and the application port are both allowed. The probe ran nslookup server.npt.svc.cluster.local followed by the same wget.

    Success conditionYou can tell a DNS-blocked egress failure from a connection-blocked one by the error text, and fix it with a DNS egress rule.

  6. Cross-namespace: a podSelector cannot see another namespace

    The last trap, and it produces a confident wrong conclusion.

    The allow-from-client policy permits podSelector: {matchLabels: {role: client}}. A Pod in a different namespace, carrying exactly that label, tries the server:

    wget: download timed out
FAILED

    Blocked. The label matches, and it does not matter.

    A bare podSelector in a from or to block is scoped to the policy's own namespace. It is not a cluster-wide label search. So a Pod in another namespace can carry every label the rule names and still be denied, which is a genuinely surprising result when you are reading the policy and the Pod side by side.

    Selecting across namespaces needs a namespaceSelector, and the combination has two forms that mean different things:

    # Pods labelled role=client, in ANY namespace labelled team=web:
- namespaceSelector: {matchLabels: {team: web}}
  podSelector: {matchLabels: {role: client}}

# EITHER any Pod in a namespace labelled team=web,
# OR a Pod labelled role=client in this namespace:
- namespaceSelector: {matchLabels: {team: web}}
- podSelector: {matchLabels: {role: client}}

    The difference is one hyphen. Two selectors in one list element are ANDed; two separate list elements are ORed. That is the most consequential piece of YAML punctuation in the NetworkPolicy API, and the second form is much more permissive than people intend.

    Selecting a namespace also needs the namespace to be selectable, and by default the only label it has is the automatic one:

    {"kubernetes.io/metadata.name":"npt-other"}

    So namespaceSelector: {matchLabels: {team: web}} matches nothing until somebody labels the namespace, and the policy that references it is valid, silent and useless. kubernetes.io/metadata.name is the label to reach for when you want one specific namespace, because it always exists.

    Worth being precise about what "selectable" means: an empty namespaceSelector: {} matches all namespaces, the same inversion as an empty podSelector. - namespaceSelector: {} in an ingress rule is "from anywhere in the cluster", which is occasionally what you want and frequently a mistake.

    bash Example session 
    kubectl logs outsider -n npt-other 2>&1 | tail -2wget: download timed outFAILEDkubectl get ns npt-other -o jsonpath='{.metadata.labels}{"\n"}'{"kubernetes.io/metadata.name":"npt-other"}

    Expected resultA Pod with the right label in the wrong namespace, blocked. It requested the fully qualified http://server.npt.svc.cluster.local/, so this is not a DNS problem.

    Success conditionYou know a bare podSelector does not cross namespaces, and which label to use to select one.

  7. Where the CNI's own view of policy lives

    One more place to look when the Kubernetes objects seem fine and traffic is still wrong: the CNI's own policy layer.

    Calico has native policy CRDs alongside the Kubernetes API, and this cluster has some already:

    calico-system   calico-system.apiserver-access         7h17m
calico-system   calico-system.default-deny             7h17m
calico-system   calico-system.kube-controller-access   7h17m

    Nobody wrote those. The Tigera operator created them to protect its own namespace, and they are invisible to kubectl get netpol, which only shows networking.k8s.io objects. On any cluster where the CNI or a service mesh installs its own policies, kubectl get netpol -A is an incomplete picture, and this is worth knowing before concluding that no policy is involved.

    Calico's native policies also support things the portable API does not: an explicit Deny action, ordering, and rules that apply cluster-wide via GlobalNetworkPolicy. A GlobalNetworkPolicy denying something is not namespaced and will not appear in any per-namespace listing.

    The reverse question, whether Calico's API shows the Kubernetes policies you wrote, has a clear answer on this install:

    No resources found in npt namespace.

    It does not. The Kubernetes NetworkPolicy objects in npt are enforced by Calico but not surfaced through its aggregated API here, so do not read an empty result as "no policy applies". Use kubectl get netpol for the Kubernetes objects and the CNI's own resources for the CNI's, and check both.

    For a specific connection, the enforcement layer itself is the last resort: Calico's per-node Felix agent renders policy into the node's packet filter, and calicoctl can evaluate which policies apply to a given endpoint. That is deeper than the CKA needs, and the Calico internals guide covers where those pieces live.

    bash Example session 
    kubectl get networkpolicies.crd.projectcalico.org -A --no-headers 2>&1 | head -3calico-system   calico-system.apiserver-access         7h17mcalico-system   calico-system.default-deny             7h17mcalico-system   calico-system.kube-controller-access   7h17mkubectl get networkpolicies.projectcalico.org -n npt 2>&1 | head -4No resources found in npt namespace.

    Expected resultCalico's own policies, which kubectl get netpol never shows, and an empty result for the Kubernetes policies through Calico's API. Both halves are the point.

    Success conditionYou know to check the CNI's own policy resources as well as networking.k8s.io ones.

Troubleshooting

Official sources

← All CKA guides