--- name: secret-management description: Manage secrets including API keys, passwords, certificates, and tokens using vault solutions like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or similar. Use when implementing secret rotation, managing certificate lifecycles, configuring secret access policies, or remediating secret exposure incidents. Triggers on phrases like "secret management", "vault", "HashiCorp Vault", "AWS Secrets Manager", "Azure Key Vault", "secret rotation", "certificate management", "API key management", "token management", "secret scanning", "secret leak", "HSM", "KMS", "encryption key", "dynamic credentials". --- # Secret Management Manage secrets including API keys, passwords, certificates, and tokens using vault solutions like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. ## Workflow ### 1. Vault Architecture ``` SECRET MANAGEMENT ARCHITECTURE ═══════════════════════════════════════ Vault Solution: HashiCorp Vault (primary) + AWS Secrets Manager (AWS-native) HashiCorp Vault Components: ═══════════════════════════════════════ → Vault Server: HA cluster (3 nodes) → Storage Backend: Consul (HA) / Vault Enterprise Storage → Listener: TLS-enabled HTTPS → Audit Device: File + Syslog → Secret Engines: · KV v2 (key-value secrets) · Database (dynamic DB credentials) · AWS (dynamic IAM credentials) · TLS (PKI certificate management) · SSH (dynamic SSH credentials) · Transits (encryption as a service) → Authentication Methods: · Kubernetes (automated for pods) · LDAP/Active Directory (human users) · AWS IAM (EC2 instances) · JWT/OIDC (external applications) · AppRole (machine-to-machine) SECRET CATEGORIES: ═══════════════════════════════════════ Category Example Rotation Storage ──────────────────────────────────────────────────────────────────── Static secrets API keys, passwords Manual Vault KV v2 Dynamic secrets DB credentials, AWS IAM Auto Vault DB/AWS Certificates TLS certs, CA certs Auto Vault PKI Encryption keys Application encryption Manual Vault Transit Tokens JWT signing keys Auto Vault ``` ### 2. Secret Rotation ``` SECRET ROTATION STRATEGY ═══════════════════════════════════════ Rotation Policy: ═══════════════════════════════════════ Secret Type Rotation Period Method Downtime ──────────────────────────────────────────────────────────────────────── Database passwords 30 days Dynamic None AWS IAM credentials 24 hours Dynamic None API keys (internal) 90 days Manual Planned API keys (external) 180 days Manual Coordinated TLS certificates 365 days Auto None JWT signing keys 365 days Auto None Application encryption 365 days Manual Planned keys DYNAMIC CREDENTIALS (Zero-Downtime): ═══════════════════════════════════════ Database (PostgreSQL) example: 1. Application requests credentials from Vault 2. Vault creates new PostgreSQL user with specific permissions 3. Vault returns username + password to application 4. Application connects using dynamic credentials 5. Credentials expire automatically (TTL: 1 hour) 6. Application renews TTL before expiry 7. On disconnect, Vault revokes credentials automatically → No static passwords stored → Credentials exist only for duration of need → Audit trail of who accessed what and when MANUAL ROTATION PROCEDURE: ═══════════════════════════════════════ Step 1: Generate new secret Step 2: Update application config (new secret) Step 3: Deploy application (rolling, zero-downtime) Step 4: Verify application connects with new secret Step 5: Revoke old secret Step 6: Document rotation in audit log Step 7: Update rotation schedule AUTOMATED ROTATION (AWS Secrets Manager): ═══════════════════════════════════════ → Lambda rotation function (AWS managed) → Schedule: Every 30 days → Steps: createSecret → setSecret → testSecret → finishSecret → Monitoring: CloudWatch alarms on rotation failure → Backup: Previous version retained (180 days) ``` ### 3. Certificate Management (PKI) ``` PKI INFRASTRUCTURE ═══════════════════════════════════════ Certificate Hierarchy: ═══════════════════════════════════════ Root CA (Vault) → Intermediate CA → Service Certificates Root CA: → Self-signed, stored offline → Validity: 10 years → Distribution: Trusted by all systems Intermediate CA: → Signed by Root CA → Validity: 5 years → Issues service certificates Service Certificates: → Signed by Intermediate CA → Validity: 365 days (auto-renewal at 30 days) → Automated issuance and renewal CERTIFICATE LIFECYCLE: ═══════════════════════════════════════ 1. Request: Application requests certificate (via Vault agent) 2. Issue: Vault generates CSR + certificate 3. Distribute: Certificate + private key to application 4. Renew: Automatic renewal at 335 days (30 days before expiry) 5. Revoke: On security incident or decommission 6. CRL/OCSP: Certificate revocation checking CERTIFICATE INVENTORY: ═══════════════════════════════════════ Certificate CN Expiry Auto-Renew Status ────────────────────────────────────────────────────────────────────────────── Web Server *.company.com 2025-03-15 Yes ✓ Active API Gateway api-gateway.internal 2025-06-20 Yes ✓ Active Service Mesh *.production.svc 2025-04-10 Yes ✓ Active mTLS Client auth-service 2025-02-28 Yes ⚠ Expires soon External Partner partner-api.external 2025-01-30 No ⚠ Manual ``` ### 4. Secret Security ``` SECRET SECURITY BEST PRACTICES ═══════════════════════════════════════ Access Control: ═══════════════════════════════════════ → Principle of least privilege → Role-based access (RBAC) in Vault → Separate policies for read/write/delete → MFA for human access to Vault Policy example (Vault HCL): path "secret/data/production/database/*" { capabilities = ["read"] } path "secret/data/production/database/app-specific" { capabilities = ["read", "list"] } path "secret/metadata/production/database/*" { capabilities = ["read", "list", "delete"] } Encryption: ═══════════════════════════════════════ → Secrets encrypted at rest (AES-256-GCM) → Transit encryption (TLS 1.3) → Transit secret engine (encryption as a service) → Reprotection (periodic re-encryption with new keys) Audit: ═══════════════════════════════════════ → All secret access logged → Audit trail includes: who, what, when, from where → Alert on: failed access, unusual patterns, bulk reads → Retention: 7 years (compliance) SECRET SCANNING: ═══════════════════════════════════════ → Pre-commit hooks: detectsecrets, gitleaks → CI pipeline: TruffleHog, git-secrets → Repository scan: GitHub secret scanning → Cloud scan: Prowler, CloudSploit → Container scan: Trivy (env vars, files) Remediation on detection: 1. REVOKE the exposed secret immediately 2. Rotate the secret (generate new one) 3. Remove from Git history (BFG Repo-Cleaner) 4. Investigate scope of exposure 5. Document and notify security team ``` ### 5. Secret Distribution ``` SECRET DISTRIBUTION PATTERNS ═══════════════════════════════════════ Pattern 1: Sidecar Agent (Kubernetes) ═══════════════════════════════════════ → Vault Agent deploys as sidecar container → Templates secrets to files on startup → Application reads secrets from file system → Auto-renewal of tokens → Auto-reload of rotated secrets Deployment manifest: spec: volumes: - name: vault-secrets emptyDir: {} initContainers: - name: vault-agent image: vault:1.15 command: [vault agent -config=vault.hcl] volumeMounts: [{name: vault-secrets, mountPath: /vault/secrets}] Pattern 2: Environment Variables ═══════════════════════════════════════ → Injected at pod startup (Kubernetes) → External Secrets Operator syncs from Vault → Stored as Kubernetes Secret → Mounted as env vars ExternalSecret manifest: apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret spec: refreshInterval: 1h secretStoreRef: {name: vault, kind: ClusterSecretStore} target: {name: app-secrets} data: - secretKey: DATABASE_PASSWORD remoteRef: {key: secret/data/production/db, property: password} Pattern 3: Direct API Call ═══════════════════════════════════════ → Application calls Vault API directly → Uses Kubernetes auth or AppRole → Retrieves secrets programmatically → Caches in memory (not disk) → Handles renewal automatically ``` ## Edge Cases - **Air-gapped**: Vault unseal without cloud connectivity - **Multi-region**: Vault replication, cross-region access - **Compliance**: PCI-DSS key management, FIPS 140-2 - **HSM**: Hardware security module for root keys - **Disaster recovery**: Vault rekey, snapshot restoration ## Integration Points - **Vault**: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager - **Kubernetes**: External Secrets Operator, Vault Agent Injector - **CI/CD**: GitHub Actions, GitLab CI (secret injection) - **Monitoring**: Prometheus Vault metrics, audit logs - **Compliance**: SIEM integration, audit logging ## Output ### Secret Management Status ``` SECRET MANAGEMENT STATUS ═══════════════════════════════════════ Secrets managed: 850 (static: 200, dynamic: 650) Certificates: 45 (all auto-renewing) Vault health: HA cluster (3/3 nodes) Rotation compliance: 95% (5 secrets overdue) Audit logs: 2.4M entries (last 30 days) Secret scans: 0 exposed secrets detected Actions: → Rotate 5 overdue secrets (by end of week) → Renew partner certificate (manual, expires Jan 30) → Review access policies for 3 decommissioned apps ```