Troubleshooting Expired Certificates

Troubleshooting / Guide 96 of 103 / Advanced

• Kubernetes1.36.4
• kubeadm1.36.4
• OpenSSL3.5.5
• Cluster1 node
• TimeAbout 34 min
• Reviewed23 August 2026

Written against the versions above. A kubeadm cluster. The expired certificate here is genuinely expired, signed with an explicit past validity window using OpenSSL 3, not simulated by changing the clock.

Before you start

• A kubeadm-built cluster and root access on a control plane node.
• The Control plane components guide, since the certificates map onto them.
• The Static Pods guide, because that is how the control plane restarts.

1. 1

   What kubeadm manages, and for how long

   kubeadm certs check-expiration is the inventory, and the first thing to run on any cluster you have inherited.

   Everything it lists is signed by one of two CAs and lasts one year from the day the cluster was built. That is the fact behind this whole guide: a kubeadm cluster left alone for twelve months stops answering.

   /etc/kubernetes/pki/ holds the certificates and keys themselves. The four .conf files beside them are kubeconfigs with a client certificate embedded in each - which is why they appear in the expiry list too, and why renewing is not only about the pki directory.

   $sudo -n kubeadm certs check-expiration 2>&1 | sed -n '/CERTIFICATE /,/^$/p' | head -12CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGEDadmin.conf                 Aug 21, 2027 13:00 UTC   363d            ca                      noapiserver                  Aug 21, 2027 13:00 UTC   363d            ca                      noapiserver-etcd-client      Aug 21, 2027 13:00 UTC   363d            etcd-ca                 noapiserver-kubelet-client   Aug 21, 2027 13:00 UTC   363d            ca                      nocontroller-manager.conf    Aug 21, 2027 13:00 UTC   363d            ca                      noetcd-healthcheck-client    Aug 21, 2027 13:00 UTC   363d            etcd-ca                 noetcd-peer                  Aug 21, 2027 13:00 UTC   363d            etcd-ca                 noetcd-server                Aug 21, 2027 13:00 UTC   363d            etcd-ca                 nofront-proxy-client         Aug 21, 2027 13:00 UTC   363d            front-proxy-ca          noscheduler.conf             Aug 21, 2027 13:00 UTC   363d            ca                      nosuper-admin.conf           Aug 21, 2027 13:00 UTC   363d            ca                      no$sudo -n ls /etc/kubernetes/pki/ | tr '\n' ' '; echoapiserver-etcd-client.crt apiserver-etcd-client.key apiserver-kubelet-client.crt apiserver-kubelet-client.key apiserver.crt apiserver.key ca.crt ca.key ca.srl etcd front-proxy-ca.crt front-proxy-ca.key front-proxy-client.crt front-proxy-client.key sa.key sa.pub$sudo -n ls /etc/kubernetes/*.conf | tr '\n' ' '; echo/etc/kubernetes/admin.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/scheduler.conf /etc/kubernetes/super-admin.conf

   Expected resultAn expiry table, the pki directory listing, and four kubeconfig files.

   Success conditionYou know what expires on this cluster and where it lives.

2. 2

   One year against ten, read from the certificates

   Do not take the table's word for it - read the certificates.

   The apiserver certificate is good for a year. The CA is good for ten, and that difference is the one to remember: kubeadm certs renew re-signs leaf certificates with a CA that is not itself being renewed. Rotating the CA is a different and much harder operation.

   The subjectAltName list is worth a look while you are here. It is why the apiserver certificate is valid for the service IP, the node name and kubernetes.default all at once - and why adding a load balancer in front of a cluster usually means re-issuing it with a new name.

   $sudo -n openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -datessubject=CN=kube-apiservernotBefore=Aug 21 12:55:16 2026 GMTnotAfter=Aug 21 13:00:16 2027 GMT$sudo -n openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -subject -datessubject=CN=kubernetesnotBefore=Aug 20 19:34:18 2026 GMTnotAfter=Aug 17 19:39:18 2036 GMT$sudo -n openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName | tail -2X509v3 Subject Alternative Name:    DNS:cka4001, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:192.168.0.191

   Expected resultA one-year apiserver certificate, a ten-year CA, and the SAN list.

   Success conditionYou can tell which certificate renewal will and will not replace.

3. 3

   The symptom: Unauthorized, saying nothing about dates

   Rather than wait a year, mint a certificate that has already expired. OpenSSL 3.5 accepts explicit validity dates, so the cluster's own CA signs a client certificate that was valid for January 2024:

   openssl x509 -req -in user.csr -CA ca.crt -CAkey ca.key -extfile ext.cnf \
     -not_before 20240101000000Z -not_after 20240201000000Z -out old.crt

   openssl x509 -checkend 0 confirms it: Certificate will expire. The identity is CN=kubernetes-admin, O=system:masters - a full cluster administrator, so nothing that follows is a permissions problem.

   Point a kubeconfig at it and this is what you get:

   error: You must be logged in to the server (Unauthorized)

   That is the whole difficulty of this failure. The message says nothing about certificates, nothing about dates, and nothing about expiry. It is identical to what a wrong CA, a missing certificate or a revoked identity would produce. kubectl version is slightly more forthcoming - it prints the client version, reaches the server, and reports the server has asked for the client to provide credentials.

   Then sign the same CSR again with normal dates. Same key, same subject, same CA - and kubectl get nodes answers. Nothing changed but the validity window, which is the proof that the dates were the problem all along.

   So the diagnostic habit worth building: when you meet Unauthorized, do not reason about RBAC. Read the certificate's dates first.

   $rm -rf /tmp/exp && mkdir -p /tmp/exp && cd /tmp/exp && openssl genrsa -out user.key 2048 2>/dev/null && openssl req -new -key user.key -out user.csr -subj "/CN=kubernetes-admin/O=system:masters" && echo "extendedKeyUsage=clientAuth" > ext.cnf && lsext.cnfuser.csruser.key$cd /tmp/exp && sudo -n openssl x509 -req -in user.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -extfile ext.cnf -not_before 20240101000000Z -not_after 20240201000000Z -out old.crt 2>&1 | tail -1subject=CN=kubernetes-admin, O=system:masters$cd /tmp/exp && openssl x509 -in old.crt -noout -subject -issuer -datessubject=CN=kubernetes-admin, O=system:mastersissuer=CN=kubernetesnotBefore=Jan  1 00:00:00 2024 GMTnotAfter=Feb  1 00:00:00 2024 GMT$cd /tmp/exp && openssl x509 -in old.crt -noout -checkend 0 || trueCertificate will expire$kubectl --kubeconfig=/tmp/exp/stale.conf get nodes 2>&1 | head -3error: You must be logged in to the server (Unauthorized)$kubectl --kubeconfig=/tmp/exp/stale.conf version 2>&1 | head -3Client Version: v1.36.4Kustomize Version: v5.8.1error: You must be logged in to the server (the server has asked for the client to provide credentials)$cd /tmp/exp && sudo -n openssl x509 -req -in user.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -extfile ext.cnf -days 365 -out new.crt 2>&1 | tail -1subject=CN=kubernetes-admin, O=system:masters$cd /tmp/exp && sudo -n chown $(id -un) new.crt && openssl x509 -in new.crt -noout -subject -datessubject=CN=kubernetes-admin, O=system:mastersnotBefore=Aug 23 01:43:05 2026 GMTnotAfter=Aug 23 01:43:05 2027 GMT$cd /tmp/exp && cp stale.conf fresh.conf && kubectl config set-credentials u --client-certificate=/tmp/exp/new.crt --client-key=/tmp/exp/user.key --embed-certs=true --kubeconfig=/tmp/exp/fresh.conf >/dev/null && kubectl --kubeconfig=/tmp/exp/fresh.conf get nodes --no-headers 2>&1 | head -2cka4001   Ready   control-plane   2d6h   v1.36.4

   Expected resultA certificate dated 2024, Certificate will expire, Unauthorized from kubectl, and the same identity working once re-signed.

   Success conditionYou have produced the exact error an expired certificate gives, and cleared it.

4. 4

   Renew everything with one command

   kubeadm certs renew all re-issues every certificate the cluster signs for itself, including the ones embedded in admin.conf, controller-manager.conf and scheduler.conf.

   The apiserver certificate's notAfter moves out a year on disk, and check-expiration reports a full 364 days across the board.

   It works even when the certificates have already expired, which is the important property when you are recovering rather than maintaining. kubeadm signs with the CA key directly - it does not authenticate to the API server to do this - so it is unaffected by the very expiry you are fixing.

   $sudo -n openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddatenotAfter=Aug 21 13:00:16 2027 GMT$sudo -n kubeadm certs renew all 2>&1 | tail -12certificate for serving the Kubernetes API renewedcertificate the apiserver uses to access etcd renewedcertificate for the API server to connect to kubelet renewedcertificate embedded in the kubeconfig file for the controller manager to use renewedcertificate for liveness probes to healthcheck etcd renewedcertificate for etcd nodes to communicate with each other renewedcertificate for serving etcd renewedcertificate for the front proxy client renewedcertificate embedded in the kubeconfig file for the scheduler manager to use renewedcertificate embedded in the kubeconfig file for the super-admin renewed Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.$sudo -n kubeadm certs check-expiration 2>&1 | sed -n '/CERTIFICATE /,/^$/p' | head -6CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGEDadmin.conf                 Aug 23, 2027 01:43 UTC   364d            ca                      noapiserver                  Aug 23, 2027 01:43 UTC   364d            ca                      noapiserver-etcd-client      Aug 23, 2027 01:43 UTC   364d            etcd-ca                 noapiserver-kubelet-client   Aug 23, 2027 01:43 UTC   364d            ca                      nocontroller-manager.conf    Aug 23, 2027 01:43 UTC   364d            ca                      no

   Expected resultA new notAfter on disk and 364 days in the expiry table.

   Success conditionEvery certificate on disk is freshly signed.

5. 5

   What the restart is actually for

   kubeadm ends with You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd. That is a safe instruction and an imprecise one, and the difference is worth measuring.

   Renew only the apiserver's serving certificate and watch the container:

   CONTAINER      CREATED             STATE     NAME             ATTEMPT
   084bcea0356db  About a minute ago  Running   kube-apiserver   0
   
   served before   notAfter=Aug 23 01:44:50 2027 GMT
   file on disk    notAfter=Aug 23 01:45:26 2027 GMT
   served after    notAfter=Aug 23 01:45:26 2027 GMT

   Same container ID. Same ATTEMPT 0. New certificate being served. The apiserver watches its serving certificate and reloads it, so no restart was involved.

   The restart is for everything that is not the serving certificate: the client certificates the apiserver presents to etcd and to kubelets, and the credentials baked into controller-manager.conf and scheduler.conf. Those are read once at startup. Skip the restart and those components keep using the old ones until they restart for some other reason - which on a quiet cluster may be never, and which is how a cluster ends up half-renewed.

   The control plane runs as static pods (guide 14), so the restart is a file move: take the manifest out of /etc/kubernetes/manifests/, wait, put it back. Expect kubectl to fail while it is gone - that is the API server being down, and it is why this belongs in a maintenance window on a single-control-plane cluster. The API came back 15 seconds after the manifest was restored.

   $sudo -n crictl ps --name kube-apiserver -o 'table' 2>/dev/null | tail -2CONTAINER           IMAGE               CREATED              STATE               NAME                ATTEMPT             POD ID              POD                      NAMESPACE084bcea0356db       b0f70fa6ec47e       About a minute ago   Running             kube-apiserver      0                   2f624d30a97e5       kube-apiserver-cka4001   kube-system$echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -enddatenotAfter=Aug 23 01:44:50 2027 GMT$sudo -n kubeadm certs renew apiserver 2>&1 | tail -2 certificate for serving the Kubernetes API renewed$sudo -n openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddatenotAfter=Aug 23 01:45:26 2027 GMT$sudo -n mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/exp/ && sleep 20 && sudo -n mv /tmp/exp/kube-apiserver.yaml /etc/kubernetes/manifests/ && echo "manifest restored"manifest restored$for i in $(seq 1 40); do kubectl get --raw='/readyz' >/dev/null 2>&1 && { echo "API server back after ${i}s"; break; }; sleep 1; doneAPI server back after 15s$kubectl get nodes --no-headerscka4001   Ready   control-plane   2d6h   v1.36.4

   Expected resultAn unchanged container serving a changed certificate, then a clean restart with the API back in about fifteen seconds.

   Success conditionYou can say which certificates reload themselves and which need the restart.

6. 6

   The kubeconfig in your home directory is a copy, and renewal misses it

   This is the part that turns a fixed cluster back into a broken one.

   check-expiration reports admin.conf at 364 days, and reading the certificate out of /etc/kubernetes/admin.conf confirms it. Now read the one your kubectl is actually using:

   /etc/kubernetes/admin.conf   notAfter=Aug 23 01:43:30 2027 GMT
   ~/.kube/config               notAfter=Aug 20 19:39:18 2027 GMT

   Two different certificates. ~/.kube/config was copied from admin.conf when the cluster was built, and kubeadm certs renew rewrote the original, not your copy. ls -l shows the copy's own date, unchanged by the renewal.

   And kubectl get nodes still works, which is exactly what makes this dangerous. The stale certificate has not expired yet - it simply expires on the original schedule. You renewed the cluster, verified it, and left your own credentials due to fail on the date you thought you had moved.

   The fix is a copy:

   sudo cp /etc/kubernetes/admin.conf ~/.kube/config
   sudo chown $(id -u):$(id -g) ~/.kube/config

   Do it every time you renew, and check any other machine holding a copy - a laptop, a CI runner, a colleague. Each is a separate copy with its own expiry, and check-expiration on the control plane knows about none of them.

   $sudo -n kubeadm certs check-expiration 2>&1 | grep -E 'admin.conf|super-admin.conf|controller-manager.conf|scheduler.conf'admin.conf                 Aug 23, 2027 01:43 UTC   364d            ca                      nocontroller-manager.conf    Aug 23, 2027 01:43 UTC   364d            ca                      noscheduler.conf             Aug 23, 2027 01:43 UTC   364d            ca                      nosuper-admin.conf           Aug 23, 2027 01:43 UTC   364d            ca                      no$sudo -n grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddatenotAfter=Aug 23 01:43:30 2027 GMT$kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddatenotAfter=Aug 20 19:39:18 2027 GMT$ls -l ~/.kube/config | awk '{print $5, $6, $7, $8, $9}'5660 Aug 22 15:11 /home/candidate/.kube/config$kubectl get nodes --no-headers 2>&1 | head -2cka4001   Ready   control-plane   2d6h   v1.36.4

   Expected resultThe renewed date on admin.conf, an older one in ~/.kube/config, and kubectl still working.

   Success conditionYou know to re-copy the kubeconfig, and why nothing warned you.

Troubleshooting

• kubectl returns Unauthorized or the server has asked for the client to provide credentials.

  Why: The client certificate was rejected during the TLS handshake, most often because it expired.

  Fix:Check the dates before checking RBAC: kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -dates. The message never mentions expiry because authentication never happened. Forbidden is the opposite situation: authentication worked and permissions did not.

• Certificates were renewed and the cluster is still broken.

  Why: The components read certificates at startup and have not restarted.

  Fix:kubeadm says so in its own output. Restart each control plane component by moving its manifest out of /etc/kubernetes/manifests and back, or restart the kubelet. Verify from the wire, not the disk: echo | openssl s_client -connect 127.0.0.1:6443 | openssl x509 -noout -dates.

• Certificates were renewed and kubectl still fails for your user.

  Why: ~/.kube/config is a copy of admin.conf made at install time. Renewal does not touch it.

  Fix:sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) ~/.kube/config. Then find every other copy: CI secrets, colleagues' machines, credential stores. Each holds its own certificate with its own expiry and none of them renews.

• kubectl cannot reach the cluster and you cannot run kubeadm through it.

  Why: Nothing. kubeadm certs renew does not need the API server.

  Fix:Renewal is a local operation using the CA keys on disk, so it works on a cluster that is refusing every connection. Run it on the control plane node, restart the components, then check kubectl. This is the normal recovery path for an expired cluster.

• x509: certificate signed by unknown authority.

  Why: The client does not trust the server's certificate. This is the reverse of an expiry problem.

  Fix:The certificate-authority-data in your kubeconfig no longer matches the cluster's CA, which happens after a rebuild or when pointing at a different cluster. Take a fresh kubeconfig from the control plane node. Renewing certificates does not cause this, because renewal keeps the same CA.

• x509: certificate has expired or is not yet valid on a certificate that looks fine.

  Why: Read the second half of that message. A wrong system clock makes valid certificates appear not-yet-valid.

  Fix:Check the time on both machines, timedatectl, before touching any certificate. Clock skew produces certificate errors that look like expiry, and it is quick to rule out.

• A new DNS name or load balancer in front of the API server fails certificate validation.

  Why: The name is not in the API server certificate's subjectAltName.

  Fix:openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -ext subjectAltName. Add the name to apiServerCertSANs in the kubeadm config, then kubeadm certs renew apiserver and restart it. A plain renew keeps the existing SANs, so the config change has to come first.

• A node that was switched off for a long time will not rejoin.

  Why: The kubelet rotates its own client certificate, but only while the current one is still valid. An expired one cannot renew itself.

  Fix:Generate a new bootstrap token on a control plane node and rejoin the node with kubeadm join. Deleting /var/lib/kubelet/pki/kubelet-client-current.pem and restarting the kubelet also works when a valid bootstrap credential is present.

• EXTERNALLY MANAGED: yes in the expiry table and kubeadm certs renew will not renew.

  Why: The CA key is not on this machine, so kubeadm cannot sign.

  Fix:The certificates come from an external PKI and that PKI must reissue them. kubeadm certs check-expiration still reports the dates, so it remains useful for monitoring even where it cannot renew.

Official sources

• Certificate management with kubeadm
• PKI certificates and requirements
• Certificate rotation for the kubelet
• Authenticating
• kubeadm certs reference