CertGrid CertGrid
Hands-on Lab·Certified Kubernetes Security Specialist

CIS Benchmark Review with kube-bench

The competency says use the CIS benchmark to review the configuration of Kubernetes components, so this guide runs it, fixes what it finds, and runs it again to prove the fix. The interesting half is the finding that stays red with the file at 600, and what that teaches about trusting a scanner.

Cluster Setup Guide 5 of 40 Intermediate

Written against the versions above. kube-bench v0.10.7 as a Job on cka8001, the single-node cluster this path uses for anything that touches the node itself. The three files it names are left hardened afterwards.

Every command on this page ran on cka8001.
Server NameIP AddressOSRolesCPURAMHDD
CKA8001192.168.0.56Ubuntu 26.04 LTSSingle Node (control plane, untainted)2 Core4 GB50 GB

Before you start

  1. Run the benchmark against the node

    A Job with the host paths the scanner needs to see.

    bash Example session 
    kubectl create ns cks-cis >/dev/null 2>&1; printf 'apiVersion: batch/v1\nkind: Job\nmetadata:\n  name: bench-before\n  namespace: cks-cis\nspec:\n  backoffLimit: 0\n  template:\n    spec:\n      hostPID: true\n      tolerations:\n      - {operator: Exists}\n      containers:\n      - name: kube-bench\n        image: docker.io/aquasec/kube-bench:v0.10.7\n        command: ["kube-bench","run","--targets=node"]\n        volumeMounts:\n        - {name: e, mountPath: /etc/kubernetes, readOnly: true}\n        - {name: v, mountPath: /var/lib, readOnly: true}\n        - {name: u, mountPath: /usr/lib/systemd, readOnly: true}\n      restartPolicy: Never\n      volumes:\n      - {name: e, hostPath: {path: /etc/kubernetes}}\n      - {name: v, hostPath: {path: /var/lib}}\n      - {name: u, hostPath: {path: /usr/lib/systemd}}\n' | kubectl apply -f - 2>&1 | tail -1; echo "--- it needs the node's own files, so it runs with host mounts. A scanner is a privileged workload and belongs under the same review as any other"job.batch/bench-before created--- it needs the node's own files, so it runs with host mounts. A scanner is a privileged workload and belongs under the same review as any otherfor i in $(seq 60); do S=$(kubectl -n cks-cis get job bench-before -o jsonpath='{.status.succeeded}' 2>/dev/null); [ "$S" = "1" ] && break; sleep 5; done; kubectl -n cks-cis logs job/bench-before 2>/dev/null | grep -A5 '== Summary node =='; kubectl -n cks-cis logs job/bench-before 2>/dev/null | grep -E '^\[FAIL\]'; echo "--- two automated failures, both file permissions. That is the usual flavour of a node finding"== Summary node ==16 checks PASS2 checks FAIL6 checks WARN0 checks INFO [FAIL] 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated)[FAIL] 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated)--- two automated failures, both file permissions. That is the usual flavour of a node finding

    Expected result16 checks PASS, 2 checks FAIL, 6 checks WARN - and both failures are file permissions, 4.1.1 and 4.1.9.

    Success conditionYou have a scored review of the node in one command.

  2. Read what it actually checked

    The two files, on the node, before touching anything.

    bash Example session 
    sudo stat -c '%a %U:%G %n' /var/lib/kubelet/config.yaml /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf; echo "--- 644, so every local user can read them. The kubelet config names the client CA and the authorization mode, which is a map of how this node decides who to trust"644 root:root /var/lib/kubelet/config.yaml644 root:root /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf--- 644, so every local user can read them. The kubelet config names the client CA and the authorization mode, which is a map of how this node decides who to trust

    Expected result644 root:root on both /var/lib/kubelet/config.yaml and /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf.

    Success conditionYou have confirmed the finding by hand.

  3. Fix the first one, and measure it

    Apply the remediation the tool printed, then run it again.

    bash Example session 
    kubectl -n cks-cis logs job/bench-before 2>/dev/null | grep -A2 '^4.1.9' | head -3; echo "--- the tool names the file and the mode. This one is exactly right"4.1.9 Run the following command (using the config file location identified in the Audit step)chmod 600 /var/lib/kubelet/config.yaml --- the tool names the file and the mode. This one is exactly rightsudo chmod 600 /var/lib/kubelet/config.yaml; sudo stat -c '%a %U:%G %n' /var/lib/kubelet/config.yaml; kubectl get nodes --no-headers; systemctl is-active kubelet; echo "--- 600 root:root, node still Ready, kubelet still active. A permission change needs no restart, which makes it the safest kind of remediation"600 root:root /var/lib/kubelet/config.yamlcka8001   Ready   control-plane   8h    v1.36.3active--- 600 root:root, node still Ready, kubelet still active. A permission change needs no restart, which makes it the safest kind of remediationprintf 'apiVersion: batch/v1\nkind: Job\nmetadata:\n  name: bench-after\n  namespace: cks-cis\nspec:\n  backoffLimit: 0\n  template:\n    spec:\n      hostPID: true\n      tolerations:\n      - {operator: Exists}\n      containers:\n      - name: kube-bench\n        image: docker.io/aquasec/kube-bench:v0.10.7\n        command: ["kube-bench","run","--targets=node"]\n        volumeMounts:\n        - {name: e, mountPath: /etc/kubernetes, readOnly: true}\n        - {name: v, mountPath: /var/lib, readOnly: true}\n        - {name: u, mountPath: /usr/lib/systemd, readOnly: true}\n      restartPolicy: Never\n      volumes:\n      - {name: e, hostPath: {path: /etc/kubernetes}}\n      - {name: v, hostPath: {path: /var/lib}}\n      - {name: u, hostPath: {path: /usr/lib/systemd}}\n' | kubectl apply -f - >/dev/null; for i in $(seq 60); do S=$(kubectl -n cks-cis get job bench-after -o jsonpath='{.status.succeeded}' 2>/dev/null); [ "$S" = "1" ] && break; sleep 5; done; kubectl -n cks-cis logs job/bench-after 2>/dev/null | grep -A5 '== Summary node =='; kubectl -n cks-cis logs job/bench-after 2>/dev/null | grep -E '^\[FAIL\]'; echo "--- 17 PASS and one failure left. One chmod, measured before and after by the same tool"== Summary node ==17 checks PASS1 checks FAIL6 checks WARN0 checks INFO [FAIL] 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated)--- 17 PASS and one failure left. One chmod, measured before and after by the same tool

    Expected resultThe remediation names the file and the mode; after chmod 600 the node is still Ready, the kubelet still active, and the rerun reports 17 checks PASS, 1 checks FAIL.

    Success conditionA finding closed, and proved closed by the same tool that raised it.

  4. The one that will not clear

    Read its remediation, then look for the file it names.

    bash Example session 
    kubectl -n cks-cis logs job/bench-after 2>/dev/null | grep -A2 '^4.1.1' | head -3; echo "--- read the path it recommends, then look for it on this machine"4.1.1 Run the below command (based on the file location on your system) on the each worker node.For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf --- read the path it recommends, then look for it on this machinels -l /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 2>&1 | tail -1; sudo stat -c '%a %U:%G %n' /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf; echo "--- the remediation names a path that does not exist here. On this distribution kubeadm installs the drop-in under /usr/lib, and that is the file the kubelet actually uses"ls: cannot access '/etc/systemd/system/kubelet.service.d/10-kubeadm.conf': No such file or directory644 root:root /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf--- the remediation names a path that does not exist here. On this distribution kubeadm installs the drop-in under /usr/lib, and that is the file the kubelet actually uses

    Expected resultThe remediation says chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, and that path does not exist - No such file or directory. The real file is under /usr/lib, at 644.

    Success conditionYou can tell a real finding from a path assumption.

  5. Harden it anyway, and watch the finding survive

    chmod the real files, then run the scan a third time.

    bash Example session 
    sudo chmod 600 /usr/lib/systemd/system/kubelet.service /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf; sudo stat -c '%a %n' /usr/lib/systemd/system/kubelet.service /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf; systemctl is-active kubelet; echo "--- both hardened to 600 and the kubelet is still active. Now run the scan a third time"600 /usr/lib/systemd/system/kubelet.service600 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.confactive--- both hardened to 600 and the kubelet is still active. Now run the scan a third timeprintf 'apiVersion: batch/v1\nkind: Job\nmetadata:\n  name: bench-third\n  namespace: cks-cis\nspec:\n  backoffLimit: 0\n  template:\n    spec:\n      hostPID: true\n      tolerations:\n      - {operator: Exists}\n      containers:\n      - name: kube-bench\n        image: docker.io/aquasec/kube-bench:v0.10.7\n        command: ["kube-bench","run","--targets=node"]\n        volumeMounts:\n        - {name: e, mountPath: /etc/kubernetes, readOnly: true}\n        - {name: v, mountPath: /var/lib, readOnly: true}\n        - {name: u, mountPath: /usr/lib/systemd, readOnly: true}\n      restartPolicy: Never\n      volumes:\n      - {name: e, hostPath: {path: /etc/kubernetes}}\n      - {name: v, hostPath: {path: /var/lib}}\n      - {name: u, hostPath: {path: /usr/lib/systemd}}\n' | kubectl apply -f - >/dev/null; for i in $(seq 60); do S=$(kubectl -n cks-cis get job bench-third -o jsonpath='{.status.succeeded}' 2>/dev/null); [ "$S" = "1" ] && break; sleep 5; done; kubectl -n cks-cis logs job/bench-third 2>/dev/null | grep -A5 '== Summary node =='; kubectl -n cks-cis logs job/bench-third 2>/dev/null | grep -E '^\[FAIL\]'; echo "--- the file is 600 and 4.1.1 STILL FAILS, because the check looks at the path it knows about and finds nothing there. The finding is about what the tool could see, not about this machine"== Summary node ==17 checks PASS1 checks FAIL6 checks WARN0 checks INFO [FAIL] 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated)--- the file is 600 and 4.1.1 STILL FAILS, because the check looks at the path it knows about and finds nothing there. The finding is about what the tool could see, not about this machine

    Expected resultBoth files at 600 and the kubelet still active - and the third run still reports 17 checks PASS, 1 checks FAIL, still 4.1.1.

    Success conditionYou know the difference between a compliant machine and a green report.

  6. The six it never assessed at all

    Read the WARN list and the word in brackets.

    bash Example session 
    kubectl -n cks-cis logs job/bench-third 2>/dev/null | grep -E '^\[WARN\]' | head -6; echo "--- every one says (Manual). A scanner cannot decide whether your kubelet arguments are appropriate, so these stay unassessed until a person writes an answer down"[WARN] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual)[WARN] 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual)[WARN] 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual)[WARN] 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual)[WARN] 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual)[WARN] 4.2.13 Ensure that a limit is set on pod PIDs (Manual)--- every one says (Manual). A scanner cannot decide whether your kubelet arguments are appropriate, so these stay unassessed until a person writes an answer down

    Expected result4.1.3, 4.1.4, 4.1.7, 4.2.9, 4.2.12 and 4.2.13 - every one marked (Manual).

    Success conditionYou will report PASS, FAIL and WARN separately.

Troubleshooting

Official sources

← All CKS guides