Secrets Management Automation

Every automated system needs credentials, and automation credentials tend to be broader in scope, longer-lived, and less reviewed than the ones humans hold. That combination makes them one of the most valuable targets in a modern estate.

The goal is not “store secrets more securely”. It is to have as few long-lived secrets as possible, ideally none.

The hierarchy of approaches link

flowchart TD
    A["Hard-coded in source<br/><small>compromised on first push</small>"] --> B["Environment variables from a config file<br/><small>visible in process listings, dumps, logs</small>"]
    B --> C["Central secret manager<br/><small>encrypted, audited, access-controlled</small>"]
    C --> D["Dynamic short-lived secrets<br/><small>generated per session, auto-expiring</small>"]
    D --> E["Workload identity<br/><small>no secret exists at all</small>"]

Each step removes a class of exposure. The last two are qualitatively different from the first three, because a credential that expires in minutes has a correspondingly small window of usefulness to an attacker, and one that never existed cannot leak.

Workload identity link

The strongest available pattern: the workload proves what it is to an identity provider, which issues a short-lived token. No secret is stored anywhere.

sequenceDiagram
    participant W as Workload (CI job / pod)
    participant I as Identity provider
    participant C as Cloud / target API
    W->>I: present platform-signed identity assertion<br/>(OIDC token from CI or the cluster)
    I->>I: verify issuer, subject, audience, claims
    I-->>W: short-lived access token (e.g. 15 min)
    W->>C: call API with the token
    C->>C: validate token, apply the mapped role
    Note over W,C: Nothing to steal from the repo,<br/>the runner, or the container

This is the correct answer for CI pipelines authenticating to cloud providers, for workloads calling cloud APIs, and increasingly for service-to-service authentication. If your CI system still holds long-lived cloud access keys as repository secrets, replacing them with OIDC federation is usually the single highest-value security change available in the pipeline.

The critical configuration detail: scope the trust condition tightly. A trust policy that accepts any token from your CI provider lets any repository on that provider assume your role. Constrain on the specific organization, repository, and where relevant the branch or environment.

Dynamic secrets link

For systems that cannot do workload identity — many databases, most legacy systems — the next best thing is a secret manager that generates credentials on demand with a short lease.

flowchart LR
    A[Application starts] --> B[Authenticate to secret manager<br/>using workload identity]
    B --> C[Request database credentials]
    C --> D[Manager creates a NEW database user<br/>with a 1-hour lease]
    D --> E[Application connects]
    E --> F{Lease expiring?}
    F -->|Yes, still needed| G[Renew] --> E
    F -->|No longer needed| H[Manager revokes the user]

The properties this gives you are worth spelling out: each instance gets a distinct credential, so an audit log identifies which workload did what; a leaked credential expires on its own; and revocation is immediate and does not require coordinating a rotation across every consumer.

Rotation link

Anything long-lived that remains must rotate automatically, because manual rotation does not happen. The pattern that avoids downtime is a two-secret overlap:

sequenceDiagram
    participant R as Rotation job
    participant S as Secret store
    participant T as Target system
    participant A as Consumers
    R->>T: create credential B (A still valid)
    R->>S: publish B as current, keep A as previous
    A->>S: consumers pick up B on next refresh
    Note over A: Overlap window — both work
    R->>R: wait for the overlap period
    R->>T: revoke credential A
    Note over R,T: Zero-downtime rotation,<br/>no coordinated restart required

This requires the target system to support two valid credentials at once — most do, via multiple API keys or multiple database users. Where it does not, rotation means a brief coordinated restart, which is a good reason to prefer dynamic secrets for that system.

Consumers must refresh secrets at runtime rather than reading them once at startup. A service that caches a secret forever turns every rotation into an outage on the next restart, whenever that happens to be.

Access control and audit link

That last row is achievable more often than people expect. If applications get secrets through workload identity, humans do not need to see production secrets in normal operation at all.

Secrets in CI/CD specifically link

Pipelines are a high-risk location because they execute contributor-supplied code with access to deployment credentials.

• Use OIDC federation rather than stored cloud keys.
• Scope secrets per environment and per stage. The test stage needs nothing that can write to production.
• Never expose secrets to workflows triggered by untrusted forks. A pull request from a fork can otherwise exfiltrate everything the workflow can read.
• Mask secrets in logs — and know that masking is a safety net, not a control, since a base64-encoded or otherwise transformed value will not match.
• Require approval for production-credential access, so a compromised pipeline definition cannot silently deploy.

When a secret leaks link

Speed matters more than tidiness. The order is fixed:

1. Rotate immediately. Before investigating, before cleaning history, before writing the incident report. The credential is compromised.
2. Revoke the old credential, do not merely replace it.
3. Audit for use. Check access logs for the leaked credential across its entire exposure window.
4. Clean the history, understanding that this does not undo the exposure.
5. Fix the control gap that let it through — usually a missing pre-commit hook or push protection.

Adoption checklist link

• CI authenticates to cloud providers via OIDC federation, not stored keys.
• OIDC trust policies are scoped to specific repositories and environments.
• Database and third-party credentials are dynamic and short-lived where supported.
• Remaining long-lived secrets rotate automatically with an overlap window.
• Applications refresh secrets at runtime rather than caching at startup.
• Access is least-privilege per workload, separated by environment, and fully audited.
• Fork-triggered workflows cannot access secrets.
• The leak response is rotate-first, and it has been rehearsed.