Kubernetes explained
Kubernetes Secrets: What They Protect and What They Do Not
A Kubernetes Secret stores small sensitive values, such as passwords and API keys, and hands them to pods as environment variables or files. By default the values are base64-encoded, not encrypted, and they sit unencrypted in the API server data store. Base64 is a way to store binary in text, not a protection. Real protection needs encryption at rest, tight RBAC, and often an external secret store.
By Muhammad Soliman, Founder at SeaGit
·
Key takeaways
- Base64 is encoding. Anyone who can read a Secret object can decode its values with one command.
- By default Secrets are stored unencrypted in etcd. Turning on encryption at rest is a separate configuration step on the API server.
- Anyone who can create a pod in a namespace can read every Secret in that namespace, even without get permission on Secrets.
- A single Secret is limited to 1 MiB. Large files belong in a volume or an external store, not in a Secret.
Base64 is not encryption, and the docs say so
The official Secrets page is blunt about the default: Secrets are stored unencrypted in etcd, and anyone with API access can retrieve or modify one. The .data field is base64 text, so a reader can decode it with a single command. The fix is a set of steps, not a different format.
# Create a Secret; the value is base64-encoded in the object
kubectl create secret generic db-creds -n prod --from-literal=password='s3cret-example'
# Anyone with get on the Secret can read it back
kubectl get secret db-creds -n prod -o jsonpath='{.data.password}' | base64 --decode; echoThe four steps the official page recommends
The Secrets page lists the minimum steps for using Secrets safely. Each one closes a different path to the value.
- Enable encryption at rest for Secrets in the API server, so etcd holds ciphertext. The encryption-at-rest task page describes the EncryptionConfiguration.
- Use RBAC with least privilege, so only the workloads and people who need a Secret can get or list it.
- Restrict which containers can use a Secret. A Secret mounted into a sidecar is readable by that sidecar too.
- Consider an external secret store, such as the Secrets Store CSI driver, so the value never lives in the cluster API as a permanent object.
Remember the namespace rule in the same page: anyone who can create a pod in a namespace can use that access to read any Secret in it. That is indirect access, and it includes creating a Deployment.
Using a Secret in a pod: environment variables and volumes
The official page says Secrets can be mounted as data volumes or exposed as environment variables. Both work, and they fail differently, so pick the one that matches how the app reads the value.
containers:
- name: api
image: registry.example.com/api:1.4.2
env:
- name: DB_PASSWORD
valueFrom:
secretKeyRef:
name: db-creds
key: password
volumeMounts:
- name: tls
mountPath: /etc/tls
readOnly: true
volumes:
- name: tls
secret:
secretName: api-tlsWhat happens when the Secret is missing
The official page says that if the Secret cannot be fetched, the kubelet keeps retrying the pod and reports an event that names the problem. The pod does not start until the Secret exists. Create the Secret before the Deployment, or the rollout waits on it.
Size, versions and what to keep out
Each Secret is capped at 1 MiB. The limit exists because very large Secrets can exhaust API server and kubelet memory. Creating many small Secrets can have the same effect, so the official page suggests a resource quota on the number of Secrets in a namespace.
- Do not commit Secret manifests to Git with real values. Use a sealed or externally managed approach if you keep manifests in a repository.
- An env var does not change when the Secret changes. Restart the pods, for example with kubectl rollout restart, to pick up a rotated value.
- Do not print Secret values into CI logs. kubectl get secret -o yaml shows the base64 values in full.
Decision rule: Secret, external store, or nothing
- Non-sensitive configuration (feature flags, log levels): use a ConfigMap, not a Secret.
- A few credentials, one cluster, encryption at rest enabled, and tight RBAC: a Kubernetes Secret is a reasonable default.
- Many environments, audited rotation, or secrets shared with non-Kubernetes systems: use an external secret manager and sync it into the cluster.