← All posts
secretssecurityrotationdevops

Secrets Rotation Is a Habit, Not a Project

Every team I audit treats secrets rotation as a one-time project. Six months later their secrets are stale again. Here is how to make it a habit instead.

10 February 2025·4 min read

Every security audit I do includes a sentence in the findings: "secrets rotation is not happening on a regular cadence". Every time. The team agrees, makes a project of rotating everything, and six months later their secrets are stale again.

The shape of the fix is the same as Kubernetes upgrades. Make it a habit, not a project, and the problem stops compounding.

Why the project approach fails

Secret rotation as a project looks like this:

  • Quarterly security review flags stale credentials.
  • Team forms a "credential rotation working group".
  • Rotates everything over six weeks of painful, error-prone work.
  • Working group dissolves.
  • Six months later, same finding in the next audit.

The rotation effort is wasted because there is no mechanism to keep it that way. The fix is not "do the project better". It is "stop having projects".

The four classes of secret

Different secrets have different rotation logics. Conflating them is part of why teams stall.

1. Machine secrets that the system should rotate itself

Database passwords for service accounts. API keys between internal services. Access tokens for cloud providers. These should be rotated by automation on a fixed cadence (30, 60, 90 days, pick one) without any human in the loop.

The pattern is well-trodden. AWS Secrets Manager handles RDS passwords. Vault handles dynamic credentials for many backends. Workload identity federation (IRSA on EKS, Workload Identity on GKE) replaces long-lived service-account keys with short-lived tokens.

If you have any service in 2025 still using a static long-lived API key from a cloud provider, that is a finding. The fix is workload identity, not "rotate the key more often".

2. Human-held secrets

SSH keys, personal access tokens, MFA seeds. Different problem entirely. Rotation here is about humans changing them, which means it is about adoption, not technology.

The right answer is to stop having human-held long-lived secrets where possible. SSO everywhere. Short-lived tokens via SSO/OIDC. SSH via a CA that signs short-lived certs. PATs replaced with OAuth flows.

Where that is not possible, age out the secrets aggressively and trust the user to renew. 90 days max for a PAT. The friction of rotation is the friction of caring about security, and it is fine.

3. Cryptographic keys

Signing keys, encryption keys, root CAs. Rotation here is high-stakes, low-frequency, and needs a different process. Annual to multi-year cadence depending on the key. The work is in the key hierarchy: the root keys rotate rarely, the leaf keys rotate often, the boundaries are clean.

If your signing infrastructure does not have this hierarchy, that is the project. Build it once. Then the leaves rotate themselves.

4. Third-party secrets

Stripe API keys. Twilio tokens. The hundred SaaS vendors your product depends on. These are the ones that get forgotten. They sit in environment variables, sometimes in source control (please not), and nobody owns rotating them.

The fix is twofold. One: catalogue them. Every third-party secret needs an owner and a rotation cadence. Two: pull them all into the same secret store the rest of your machine secrets live in. From there, rotation is at least manual but uniform. From there, you can build automation per vendor.

The habits that actually work

Three practices that move teams from "rotation is a project" to "rotation is a habit":

Burn-in alarms

For every secret in your store, attach an alarm that fires when the secret has not rotated in N days. N depends on the class. The alarm goes to the team that owns the secret. The alarm is a paging-priority alert, not an email.

Most teams have these for certificates and nothing else. The same logic should apply to every credential of consequence.

Secret store as the only source of truth

If a service reads a secret from anywhere other than the secret store at runtime, it is a problem. Environment variables baked into images are problems. Files mounted from a VCS are problems. The store has to be the only path, because that is the only place rotation is feasible.

This is more architectural work than people think. It is also the unblocker for everything else.

Drill it like an incident

Once a quarter, pick a high-impact secret, rotate it manually, and time how long the rotation takes from "decide to rotate" to "old credential is dead and removed from everywhere". The first time you do this it will take days and break things. Each subsequent drill will be faster.

The drill exposes the rotation paths that do not work. That is the point. You are not actually responding to a compromise; you are practising for one.

The cultural part

Rotation as a habit only works if leadership treats it as a baseline, not a heroic effort. The way I sell this to clients: track two metrics on the security dashboard. Median age of credentials in the secret store. Number of credentials older than the rotation policy.

Both should be small and trending small. If the median age is climbing, the team has lost the habit, and that is a leading indicator of the next breach. The metric is the discipline.

Where to start

If you are reading this and your secrets rotation is currently a project that someone schedules every other quarter, do this in order:

  1. Inventory. You probably have more long-lived secrets than you think.
  2. Classify. Machine, human, cryptographic, third-party.
  3. Eliminate the long-lived ones you can replace with workload identity or SSO.
  4. Rotate the rest on a calendar.
  5. Drill quarterly.

It is not glamorous. It is the difference between an organisation that has a small ongoing security cost and one that has a periodic catastrophic one.