Take backup of all sealed-secrets keys or re-encrypt secrets regularly

By default, sealed-secret certificates get automatically renewed every 30 days. And kubeseal uses the latest key to encrypt new secrets.

But sealed secrets are not automatically rotated, and old keys are not deleted when new keys are generated. Old sealed secrets resources can be still decrypted.

All these details are good as long as you use single k8s cluster! But let’s think about that your cluster is entirely not reachable and you store secrets in git repository encrypted with kubeseal.So you don’t have access to plain secrets.

Then you are in trouble, you have to re-create all secrets in your new cluster with from scratch and update deployments.

This is worst-case scenario! But you should prepare yourself for this case.

Possible solutions:

Solution #1

Take backup of all sealed-secrets keys regularly and store in a secure place and restore it when you need in new cluster. You can use the following command to dump all active sealed secret keys in kube-system namespace.

$ kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml > allsealkeys.yml

PS: You can use kubectl neat plugin to get rid of unnecessary lines in yaml output

$ kubectl -n kube-system get secret -l sealedsecrets.bitnami.com/sealed-secrets-key=active -o yaml | kubectl neat > allsealkeys.yml

Solution #2

Use your own generated certificate and disable key rotation by setting key-renew-period=0

But this is a bit less secure way.

Solution #3

Re-encrypt your existing secrets whenever sealed-secrets create new certificates. So your secrets in git repository will be encrypted with the latest keys and just take the backup of the latest key. See https://github.com/bitnami-labs/sealed-secrets#re-encryption-advanced for more details.

Ismail YENIGUL

DevSecOps Engineer