IT AI Skill

Secret Management

Manage secrets including API keys, passwords, certificates, and tokens using vault solutions like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or similar. Use when implementing secret rotation, managing certificate lifecycles, configuring secret a...

Secret Management

Manage secrets including API keys, passwords, certificates, and tokens using vault solutions like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault.

Workflow

1. Vault Architecture

SECRET MANAGEMENT ARCHITECTURE
═══════════════════════════════════════

Vault Solution: HashiCorp Vault (primary) + AWS Secrets Manager (AWS-native)

HashiCorp Vault Components:
═══════════════════════════════════════

  → Vault Server: HA cluster (3 nodes)
  → Storage Backend: Consul (HA) / Vault Enterprise Storage
  → Listener: TLS-enabled HTTPS
  → Audit Device: File + Syslog
  → Secret Engines:
     · KV v2 (key-value secrets)
     · Database (dynamic DB credentials)
     · AWS (dynamic IAM credentials)
     · TLS (PKI certificate management)
     · SSH (dynamic SSH credentials)
     · Transits (encryption as a service)

  → Authentication Methods:
     · Kubernetes (automated for pods)
     · LDAP/Active Directory (human users)
     · AWS IAM (EC2 instances)
     · JWT/OIDC (external applications)
     · AppRole (machine-to-machine)

SECRET CATEGORIES:
═══════════════════════════════════════

Category            Example                    Rotation    Storage
────────────────────────────────────────────────────────────────────
Static secrets      API keys, passwords        Manual      Vault KV v2
Dynamic secrets     DB credentials, AWS IAM    Auto        Vault DB/AWS
Certificates        TLS certs, CA certs        Auto        Vault PKI
Encryption keys     Application encryption     Manual      Vault Transit
Tokens              JWT signing keys           Auto        Vault

2. Secret Rotation

SECRET ROTATION STRATEGY
═══════════════════════════════════════

Rotation Policy:
═══════════════════════════════════════

Secret Type              Rotation Period    Method          Downtime
────────────────────────────────────────────────────────────────────────
Database passwords       30 days            Dynamic         None
AWS IAM credentials      24 hours           Dynamic         None
API keys (internal)      90 days            Manual          Planned
API keys (external)      180 days           Manual          Coordinated
TLS certificates         365 days           Auto            None
JWT signing keys         365 days           Auto            None
Application encryption   365 days           Manual          Planned
keys

DYNAMIC CREDENTIALS (Zero-Downtime):
═══════════════════════════════════════

  Database (PostgreSQL) example:
  1. Application requests credentials from Vault
  2. Vault creates new PostgreSQL user with specific permissions
  3. Vault returns username + password to application
  4. Application connects using dynamic credentials
  5. Credentials expire automatically (TTL: 1 hour)
  6. Application renews TTL before expiry
  7. On disconnect, Vault revokes credentials automatically

  → No static passwords stored
  → Credentials exist only for duration of need
  → Audit trail of who accessed what and when

MANUAL ROTATION PROCEDURE:
═══════════════════════════════════════

  Step 1: Generate new secret
  Step 2: Update application config (new secret)
  Step 3: Deploy application (rolling, zero-downtime)
  Step 4: Verify application connects with new secret
  Step 5: Revoke old secret
  Step 6: Document rotation in audit log
  Step 7: Update rotation schedule

AUTOMATED ROTATION (AWS Secrets Manager):
═══════════════════════════════════════

  → Lambda rotation function (AWS managed)
  → Schedule: Every 30 days
  → Steps: createSecret → setSecret → testSecret → finishSecret
  → Monitoring: CloudWatch alarms on rotation failure
  → Backup: Previous version retained (180 days)

3. Certificate Management (PKI)

PKI INFRASTRUCTURE
═══════════════════════════════════════

Certificate Hierarchy:
═══════════════════════════════════════

  Root CA (Vault) → Intermediate CA → Service Certificates

  Root CA:
    → Self-signed, stored offline
    → Validity: 10 years
    → Distribution: Trusted by all systems

  Intermediate CA:
    → Signed by Root CA
    → Validity: 5 years
    → Issues service certificates

  Service Certificates:
    → Signed by Intermediate CA
    → Validity: 365 days (auto-renewal at 30 days)
    → Automated issuance and renewal

CERTIFICATE LIFECYCLE:
═══════════════════════════════════════

  1. Request: Application requests certificate (via Vault agent)
  2. Issue: Vault generates CSR + certificate
  3. Distribute: Certificate + private key to application
  4. Renew: Automatic renewal at 335 days (30 days before expiry)
  5. Revoke: On security incident or decommission
  6. CRL/OCSP: Certificate revocation checking

CERTIFICATE INVENTORY:
═══════════════════════════════════════

Certificate           CN                        Expiry     Auto-Renew   Status
──────────────────────────────────────────────────────────────────────────────
Web Server            *.company.com            2025-03-15   Yes          ✓ Active
API Gateway           api-gateway.internal     2025-06-20   Yes          ✓ Active
Service Mesh          *.production.svc         2025-04-10   Yes          ✓ Active
mTLS Client           auth-service             2025-02-28   Yes          ⚠ Expires soon
External Partner      partner-api.external     2025-01-30   No           ⚠ Manual

4. Secret Security

SECRET SECURITY BEST PRACTICES
═══════════════════════════════════════

Access Control:
═══════════════════════════════════════

  → Principle of least privilege
  → Role-based access (RBAC) in Vault
  → Separate policies for read/write/delete
  → MFA for human access to Vault

  Policy example (Vault HCL):
    path "secret/data/production/database/*" {
      capabilities = ["read"]
    }
    path "secret/data/production/database/app-specific" {
      capabilities = ["read", "list"]
    }
    path "secret/metadata/production/database/*" {
      capabilities = ["read", "list", "delete"]
    }

Encryption:
═══════════════════════════════════════

  → Secrets encrypted at rest (AES-256-GCM)
  → Transit encryption (TLS 1.3)
  → Transit secret engine (encryption as a service)
  → Reprotection (periodic re-encryption with new keys)

Audit:
═══════════════════════════════════════

  → All secret access logged
  → Audit trail includes: who, what, when, from where
  → Alert on: failed access, unusual patterns, bulk reads
  → Retention: 7 years (compliance)

SECRET SCANNING:
═══════════════════════════════════════

  → Pre-commit hooks: detectsecrets, gitleaks
  → CI pipeline: TruffleHog, git-secrets
  → Repository scan: GitHub secret scanning
  → Cloud scan: Prowler, CloudSploit
  → Container scan: Trivy (env vars, files)

  Remediation on detection:
    1. REVOKE the exposed secret immediately
    2. Rotate the secret (generate new one)
    3. Remove from Git history (BFG Repo-Cleaner)
    4. Investigate scope of exposure
    5. Document and notify security team

5. Secret Distribution

SECRET DISTRIBUTION PATTERNS
═══════════════════════════════════════

Pattern 1: Sidecar Agent (Kubernetes)
═══════════════════════════════════════

  → Vault Agent deploys as sidecar container
  → Templates secrets to files on startup
  → Application reads secrets from file system
  → Auto-renewal of tokens
  → Auto-reload of rotated secrets

  Deployment manifest:
    spec:
      volumes:
      - name: vault-secrets
        emptyDir: {}
      initContainers:
      - name: vault-agent
        image: vault:1.15
        command: [vault agent -config=vault.hcl]
        volumeMounts: [{name: vault-secrets, mountPath: /vault/secrets}]

Pattern 2: Environment Variables
═══════════════════════════════════════

  → Injected at pod startup (Kubernetes)
  → External Secrets Operator syncs from Vault
  → Stored as Kubernetes Secret
  → Mounted as env vars

  ExternalSecret manifest:
    apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    spec:
      refreshInterval: 1h
      secretStoreRef: {name: vault, kind: ClusterSecretStore}
      target: {name: app-secrets}
      data:
      - secretKey: DATABASE_PASSWORD
        remoteRef: {key: secret/data/production/db, property: password}

Pattern 3: Direct API Call
═══════════════════════════════════════

  → Application calls Vault API directly
  → Uses Kubernetes auth or AppRole
  → Retrieves secrets programmatically
  → Caches in memory (not disk)
  → Handles renewal automatically

Edge Cases

Air-gapped: Vault unseal without cloud connectivity
Multi-region: Vault replication, cross-region access
Compliance: PCI-DSS key management, FIPS 140-2
HSM: Hardware security module for root keys
Disaster recovery: Vault rekey, snapshot restoration

Integration Points

Vault: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
Kubernetes: External Secrets Operator, Vault Agent Injector
CI/CD: GitHub Actions, GitLab CI (secret injection)
Monitoring: Prometheus Vault metrics, audit logs
Compliance: SIEM integration, audit logging

Output

Secret Management Status

SECRET MANAGEMENT STATUS
═══════════════════════════════════════

Secrets managed: 850 (static: 200, dynamic: 650)
Certificates: 45 (all auto-renewing)
Vault health: HA cluster (3/3 nodes)
Rotation compliance: 95% (5 secrets overdue)
Audit logs: 2.4M entries (last 30 days)
Secret scans: 0 exposed secrets detected

Actions:
  → Rotate 5 overdue secrets (by end of week)
  → Renew partner certificate (manual, expires Jan 30)
  → Review access policies for 3 decommissioned apps