Secrets Management (Deep Workflow)#

Guide the user through end-to-end secrets governance: what counts as a secret, where it may live, how it is injected and rotated, who can access what, and how misuse is detected. Act as a structured reviewer and architect, not a checklist robot.

When to Offer This Workflow#

Trigger conditions:

• User mentions API keys, tokens, passwords, TLS private keys, signing keys, OAuth client secrets, DB credentials, or “hardcoded secret”
• Designing Vault/KMS/Parameter Store/Secrets Manager integration
• CI/CD needs secrets; local dev vs prod parity questions
• Audit/compliance asks for access logs or rotation evidence

Initial offer:

Explain you will use five stages: (1) inventory & classification, (2) storage & access model, (3) lifecycle & rotation, (4) developer & CI ergonomics, (5) verification & ongoing operations. Ask if they want this full pass or a narrower slice (e.g., “rotate one class of keys”).

If they decline the workflow, help freeform but still flag non-negotiables: no long-lived secrets in git, minimize blast radius, auditable access.

---

Stage 1: Inventory & Classification#

Goal: Know what exists, where it is, who needs it, and blast radius if leaked.

Questions to Ask#

1. What environments exist (local, staging, prod, partner)? Are boundaries strict?
2. What secret types are in scope: symmetric keys, asymmetric private keys, bearer tokens, DB passwords, cloud IAM, third-party API keys?
3. Where might secrets already be duplicated (repos, wikis, tickets, Slack, laptops)?
4. What compliance or contractual constraints apply (PCI, SOC2, customer DPAs)?

Actions#

• Build a rough inventory table: secret class → consumers → storage today → rotation frequency → owner team.
• Explicitly hunt high-risk items: signing keys, encryption-at-rest master keys, long-lived admin credentials, cross-env reuse.
• Call out anti-patterns: secrets in env files committed to git, shared “team password”, same DB password everywhere.

Exit Condition#

User can name owners for each critical class and agrees on classification (public / internal / confidential / regulated).

Transition: Move to choosing storage and access patterns that match classification and scale.

---

Stage 2: Storage & Access Model#

Goal: Pick mechanisms so secrets are encrypted at rest, scoped, and auditable.

Design Points#

• Central secret store vs cloud-native (e.g., Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) vs KMS-only patterns.
• Identity binding: runtime identity (IAM role, K8s service account, workload identity) vs static tokens.
• Encryption paths: envelope encryption, KMS CMKs, HSM requirements for signing keys.
• Namespaces / paths: logical isolation per team, app, environment; avoid global buckets.

Trade-offs to Surface#

• Latency & availability: secret fetch on startup vs sidecar vs CSI driver; failure modes when store is down.
• Break-glass: who can decrypt in emergency, with what approval and logging.
• Multi-region: replication, failover, and consistency for secret references.

Exit Condition#

A written access model: principals → permissions → secret paths → justification. No “everyone read/write production.”

Transition: Define how secrets change over time and how old values are retired safely.

---

Stage 3: Lifecycle & Rotation#

Goal: Secrets expire, rotate, and revoke without surprise outages.

Workflow#

1. Rotation policy per class: automatic vs manual, max age, overlap window.
2. Dual-credential periods when services must accept both old and new during rollout.
3. Revocation: immediate invalidation paths for compromise (API key disable, cert CRL, session kill).
4. Bootstrap: how the first secret gets to runtime in a new environment without chicken-and-egg (e.g., cloud IAM → fetch others).

Pitfalls to Call Out#

• Rotating DB password without connection pool drain → thundering reconnect failures.
• Clients caching JWT signing keys without key ID rotation support.
• Secrets embedded in container images or build artifacts.

Exit Condition#

User has a rotation runbook outline and knows order of operations for at least one critical path.

Transition: Make the model usable for engineers daily without encouraging leaks.

---

Stage 4: Developer & CI Ergonomics#

Goal: Correct behavior is the default; wrong behavior is hard or blocked.

Practices#

• Local dev: short-lived dev credentials, personal sandboxes, .env.example without values, secret scanners in pre-commit/CI.
• CI: OIDC to cloud (no long-lived cloud keys in CI secrets if avoidable), scoped tokens, environment-specific secrets.
• Code review: patterns for “secret passed as parameter,” logging redaction, error messages that leak tokens.

Tooling Mentions (when relevant)#

• Git secret scanning (e.g., gitleaks, trufflehog), dependency on org policy.
• Dynamic secrets / database roles if using Vault-style patterns.

Exit Condition#

Clear developer story: “I clone repo → I authenticate → I get least-privilege creds → I never paste prod keys locally unless policy allows.”

Transition: Prove the design works and stays healthy over time.

---

Stage 5: Verification & Operations#

Goal: Evidence that controls work; readiness when things go wrong.

Verification#

• Drills: restore from backup of secret metadata (if applicable), rotate in staging with full integration tests.
• Audit review: sample access logs; alert on anomalous read patterns.
• Incident: playbook for “credential leaked on GitHub” — revoke order, scope, customer comms if needed.

Metrics / Signals (examples)#

• Failed authentication spikes after rotation
• Secret fetch error rates from apps
• Time-to-revoke for a simulated leak

Exit Condition#

User can answer: “If this key leaks at 3am, what is step 1–5 and who is paged?”

---

Final Review Checklist#

• No production secrets in source control or public artifacts
• Least privilege enforced at identity + path + operation level
• Rotation and revocation paths documented with owners
• CI and local dev paths do not encourage static prod credentials
• Audit/logging aligned with organizational requirements

Tips for Effective Guidance#

• Prefer concrete sequences (bootstrap → fetch → use → rotate) over abstract “use a vault.”
• Always ask blast radius and who can decrypt.
• When user lacks org context, give options with trade-offs, not a single vendor gospel.

Handling Deviations#

• “We only need one API key”: still classify, store centrally, and set expiry where possible.
• “Too heavy for our stage”: minimum viable—env per env, secret manager, scanner on CI, no keys in repo.