7 Common Kubernetes Secrets Management Mistakes and How to Fix Them

A Secret in Kubernetes is a promise that a value will be handled carefully. Nothing in the platform enforces that promise. The object is only as safe as the RBAC around it, the storage underneath it, and the way your workloads consume it. Most credential leaks we see start with something small and boring: a value pasted into a Deployment during an incident, a wildcard role binding added for debugging and never removed, a cloud key that has been valid longer than the cluster has existed. Here are the seven mistakes that come up again and again, and what to do about each.

1. Hardcoding Secrets in Manifests, Helm Values, and Images

A password typed into an env: block works on the first try, which is exactly why it spreads. That value then lands in your Git history, in the last-applied-configuration annotation, inside Helm release metadata, and in every copy of the manifest on a teammate's laptop. Bake it into an image and it survives in the layer history permanently, even if the next commit removes the line.

Rotate first, clean up second. The credential is what leaked, not the file, and deleting a commit does nothing about forks, CI caches or clones. Then move values out of the manifest: an external store such as Vault or a cloud secret manager read through External Secrets Operator or the Secrets Store CSI driver, or SOPS-encrypted values committed deliberately with the key held in KMS. Add a pre-commit hook that scans for high-entropy strings and blocks the push.

2. The Pipeline Leaks Secrets Before the Cluster Ever Sees Them

Pipelines are where secrets quietly become public. Build arguments show up in image metadata. A stray set -x prints a token into a log that fifty people can read. Jobs triggered by pull requests from forks may still receive your environment secrets unless you explicitly deny them. Artifacts and caches get shared between branches that should never see each other's credentials.

The strongest fix is to stop storing long-lived keys in CI at all. Use OIDC federation so the runner exchanges a short-lived identity token for cloud credentials, scoped to a branch or protected environment. Mask variables, restrict secret access to protected refs, and never let untrusted PRs run jobs that carry production credentials. Treat logs as public output and assume anything printed is already exposed.

3. Treating Base64 Encoding as Encryption

Run kubectl get secret my-app -o jsonpath='{.data.password}' | base64 -d and the password appears in your terminal. That is the whole design: Secret values are base64-encoded, not encrypted. Anyone with get on that Secret reads it in plaintext, and so does anyone holding an etcd snapshot, a cluster backup, or a restored copy of a namespace.

So treat secrets as plaintext that happens to be tidy. Protect them with RBAC, encrypt them at rest, and keep them out of anything that gets copied around: no secrets in Terraform state without encryption, no secrets in ticket attachments, no secrets in Slack threads during an outage. If a value appears in a backup file, that file is now a credential.

4. Overly Broad RBAC on Secrets

This is the most common and the most damaging. Look for these patterns:

• A CI or monitoring service account bound to cluster-admin because it was easier than writing a Role.
• Roles granting verbs: ["*"] on resources: ["secrets"].
• get, list, watch on secrets across every namespace.
• Workloads running under the default service account with an automounted token they never use.

Bind permissions per namespace, and use resourceNames when a workload only needs one Secret. Be aware that resourceNames does not filter list or watch, so a role that needs a single Secret should grant get only. Audit regularly with kubectl auth can-i --list --as=system:serviceaccount:namespace:name, and set automountServiceAccountToken: false as the default so tokens must be requested deliberately.

5. Leaving etcd Unencrypted

Without an encryption provider configured, Secret values sit in etcd essentially as readable text. Anyone who can reach etcd, or who obtains a snapshot file, has every credential in the cluster. Fixing this is a control-plane configuration change: define an EncryptionConfiguration with an aescbc, secretbox or KMS provider, pass it to the API server with --encryption-provider-config, then rewrite existing Secrets so old plaintext copies are replaced.

Verify rather than assume: reading a secret key directly from etcd should show a k8s:enc: prefix. Plan for key rotation and store the encryption key somewhere other than the etcd backup, because a backup without its key is worthless and a key stored beside it is worthless protection.

6. Mounting Secrets as Environment Variables Everywhere

Environment variables are convenient and leaky. They appear in /proc/1/environ, get inherited by child processes, show up in crash dumps and debug sidecars, and are visible to anyone who can exec into the pod. They also cannot be updated without a restart.

Mount secrets as files instead, with defaultMode: 0400, and have the application read the file at startup or watch it for changes. Project service account tokens with an explicit audience and short expiry rather than relying on long-lived static tokens. If a container never talks to the API, disable token mounting entirely.

7. Credentials That Never Rotate and Access Nobody Audits

A secret created three years ago and shared by four teams in two namespaces has no owner and no expiry. Prefer short-lived credentials issued through cloud workload identity so rotation happens automatically. Where static secrets are unavoidable, rotate them on a schedule from a job, keep an overlap window so deploys do not break, and label each Secret with an owner and a rotation date.

Turn on audit logging and watch secret reads. Keep the audit level at Metadata for secrets — RequestResponse records the secret body into your log store, which creates a new problem. Alert on reads from service accounts that have no business reading them, and on sudden spikes in list secrets calls.

A Baseline You Can Apply This Week

You do not need a platform project to make meaningful progress. Work in this order:

1. Scan your Git history for credentials and rotate everything the scan finds.
2. Replace long-lived cloud keys with OIDC and workload identity.
3. Enable encryption at rest and confirm it by reading from etcd.
4. Review every binding that can read secrets and delete the wildcards.
5. Convert environment-variable secrets to read-only file mounts.
6. Set token automounting to off by default and opt in per workload.
7. Add owner and rotation labels, then put rotation on a calendar.

Start with whichever item removes the most privilege fastest — usually the RBAC review. Then make the safer path the easier path, so the next person under pressure at 2 a.m. does not have to choose between shipping the fix and protecting the credential.

Photo: neelam279 / Pixabay