CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Administrator

Troubleshooting RBAC Forbidden Errors

Three Forbidden errors from one ServiceAccount, differing by verb, by resource and by namespace. Each names every field you need, and they map one-to-one onto the lines of a Role you have not written yet.

Troubleshooting Guide 102 of 103 Intermediate

Written against the versions above. The message format is stable across versions and is the primary diagnostic. Read it rather than guessing.

Any cluster with RBAC enabled, which is any kubeadm cluster.
Server NameIP AddressOSRolesCPURAMHDD
CKA1001192.168.0.175Ubuntu 26.04 LTSControl Plane Node2 Core4 GB50 GB
CKA1001-NODE01192.168.0.176Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE02192.168.0.177Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB
CKA1001-NODE03192.168.0.178Ubuntu 26.04 LTSWorker Node2 Core4 GB50 GB

Before you start

  1. A deliberately narrow ServiceAccount

    Four commands produce an identity that can do exactly one thing: get and list Pods in t3.

    auth can-i confirms both sides before anything is tested for real: yes for listing Pods, no for listing Deployments. That is the right habit, because it answers the question without needing to run the failing command and interpret its output.

    --as=system:serviceaccount:t3:limited is impersonation. Note the format, which is fixed: system:serviceaccount::. That string is the username the API server sees, and it is what will appear in every error message and audit log entry for this identity.

    Impersonation itself is a privileged operation and requires the impersonate verb, so this works as admin and would not as an ordinary user. It is the single most useful RBAC debugging tool available, because it lets you reproduce someone else's permission failure from your own terminal without their credentials.

    bash Example session 
    kubectl create serviceaccount limited -n t3serviceaccount/limited createdkubectl create role pod-lister --verb=get,list --resource=pods -n t3role.rbac.authorization.k8s.io/pod-lister createdkubectl create rolebinding limited-lists -n t3 --role=pod-lister --serviceaccount=t3:limitedrolebinding.rbac.authorization.k8s.io/limited-lists createdkubectl auth can-i list pods -n t3 --as=system:serviceaccount:t3:limitedyeskubectl auth can-i list deployments -n t3 --as=system:serviceaccount:t3:limitednokubectl get pods -n t3 --as=system:serviceaccount:t3:limited --no-headers | wc -l4

    Expected resultyes, no, and four Pods actually listed under impersonation. The permitted operation genuinely works, which matters: this is not a mock.

    Success conditionThe ServiceAccount can list Pods and not Deployments.

  2. Three failures, three different fields

    Now the errors, and the point of the guide. Read them side by side:

    deployments.apps is forbidden: User "system:serviceaccount:t3:limited"
  cannot list resource "deployments" in API group "apps" in the namespace "t3"

pods "client" is forbidden: User "system:serviceaccount:t3:limited"
  cannot delete resource "pods" in API group "" in the namespace "t3"

pods is forbidden: User "system:serviceaccount:t3:limited"
  cannot list resource "pods" in API group "" in the namespace "default"

    Same identity, three different reasons, and each message names all five fields that decide an RBAC verdict:

    1. User - the identity. If this is not who you expected, the problem is the Pod's serviceAccountName, not RBAC.
    2. verb - list, delete. The Role grants get and list only, so delete fails.
    3. resource - deployments, pods.
    4. API group - apps for Deployments, "" for Pods. That empty pair of quotes is the core group, and it is the field people get wrong most often.
    5. namespace - t3 versus default. The third error is the same verb on the same resource, failing purely because a RoleBinding's reach stops at its own namespace.

    Those map directly onto a Role. To permit the second error you add delete to verbs; the third needs a whole new RoleBinding in default.

    So the message is not a generic refusal. It is a specification of the rule you are missing, and reading it carefully is faster than any amount of experimenting.

    One detail in the second error worth noticing: pods "client" is forbidden names the object, because delete targets a specific resource. The list errors have no name because listing has no single target. When a message names an object, resourceNames in a Role can restrict access to exactly that one, which is occasionally what you want.

    bash Example session 
    kubectl get deployments -n t3 --as=system:serviceaccount:t3:limited 2>&1 | tail -2Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:t3:limited" cannot list resource "deployments" in API group "apps" in the namespace "t3"kubectl delete pod client -n t3 --as=system:serviceaccount:t3:limited 2>&1 | tail -2Error from server (Forbidden): pods "client" is forbidden: User "system:serviceaccount:t3:limited" cannot delete resource "pods" in API group "" in the namespace "t3"kubectl get pods -n default --as=system:serviceaccount:t3:limited 2>&1 | tail -2Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:t3:limited" cannot list resource "pods" in API group "" in the namespace "default"

    Expected resultThree Forbidden errors differing in exactly one field each. Note the third is a failed delete that did not happen: impersonating a restricted identity is a safe way to test destructive permissions, since the operation is refused.

    Success conditionYou can name which field differs in each of the three messages.

  3. What the identity can do, in full

    auth can-i --list inverts the question. Rather than testing one operation, it enumerates everything the identity is permitted, which is what you want when inheriting a cluster or auditing an account.

    Filtered to the interesting line:

    pods    []    []    [get list]

    The two empty brackets are Non-Resource URLs and Resource Names, both unrestricted within what is granted. [get list] is the verb set, matching the Role exactly.

    Three practical notes on using this command.

    Filter it. The unfiltered output includes permissions every authenticated identity has (selfsubjectaccessreviews, so it can run auth can-i about itself) plus anything operator-installed ClusterRoles grant. Reading it raw and being surprised is normal; grep for the resource you care about.

    It is namespace-scoped. --list -n t3 shows permissions in t3. The same identity may have different permissions elsewhere, and cluster-scoped grants appear separately.

    It answers about the authoriser's decision, not about your Roles. So it accounts for every binding that applies, including ones in other namespaces and ClusterRoleBindings you did not know about. That makes it authoritative in a way that reading Role YAML is not.

    The workflow this all adds up to, when someone reports a permission problem:

    1. Read the Forbidden message and note the five fields.
    2. Confirm with auth can-i -n --as= that you can reproduce it.
    3. auth can-i --list --as= to see what they do have.
    4. Add the missing rule, and re-test with step 2 before telling them to try again.

    Step 2 is the one people skip, and it is the one that catches the case where the identity in the message is not the identity you assumed.

    bash Example session 
    kubectl auth can-i --list -n t3 --as=system:serviceaccount:t3:limited | grep -E 'Resources|^pods'Resources                                       Non-Resource URLs                      Resource Names   Verbspods                                            []                                     []               [get list]

    Expected resultOne line matching the Role that was created. The filter is deliberate: the unfiltered list on this cluster also contains Calico CRD permissions from an operator-installed ClusterRole, which are not part of what was granted here.

    Success conditionauth can-i --list shows pods [get list] and nothing more for that resource.

Troubleshooting

Official sources

← All CKA guides