It happened twice in the last two months. Two times I tried to login to my k3s instance using a local kubectl installation and it failed due to authorization issues. Both times I had to search for my ssh keys (which I keep locked away for good reasons) to restore my ability to login to my cluster.
The underlying issue is that k3s has a sane default validity for certificates of 365 days and will renew the certs automatically at startup within 90 days prior to expiration. This means that every client certificate that is used to authenticate local kubectl access for example, also expires after 365 days. However, those certificates live in your local configuration file and can’t be automatically renewed. So if the renewal date comes and you forget to refresh your local config, you won’t be able to login. The first time this happened to me, I did not know what was going on and tried a bunch of different things before I understood what was going on.
Today it happened again and I did not immediately realize that I already encountered this issue once. When I learn something new, I try to create a note in my obsidian vault to be able to recall that knowledge easily. I did that for this issue and it proved helpful as I was able to get back up and running in less than 10 minutes. But this issue was completely preventable. I should’ve updated all certificates on all of my machines to make sure this won’t happen again soon. But I was lazy and didn’t do it. So take this as a reminder to deal with issues properly, even if it involves more work, you’ll be thankful later. As for me, I’m setting reminders to refresh my k3s certificates (yes, on all machines) before they expire and try to remember this incident to help overcome my laziness.
0 Comments