IT AI Skill

Devsecops Pipeline Security

Integrate security into CI/CD pipelines with SAST, DAST, SCA, container scanning, secrets detection, IaC scanning, software bill of materials (SBOM), security gating, and shift-left practices. Use when securing CI/CD pipelines, implementing shift-left secur...

DevSecOps Pipeline Security

Embed security testing and compliance into every stage of the CI/CD pipeline without slowing delivery.

Workflow

  1. Define security requirements per pipeline: scan types, quality gates, compliance checks, approved libraries.
  2. Configure pre-commit hooks: secrets detection, basic linting, license compliance checks.
  3. Implement build-time security: SAST, SCA/dependency scanning, license checks, IaC scanning.
  4. Add pre-deployment security: DAST, container image scanning, runtime security baseline, SBOM generation.
  5. Enable post-deployment security: RASP, continuous monitoring, vulnerability re-scanning, compliance drift detection.
  6. Enforce security gates: fail builds on critical/high vulnerabilities, block deployments on policy violations.
  7. Track and report: security metrics, vulnerability trends, mean time to remediate, compliance scores.
  8. Continuously improve: reduce false positives, tune thresholds, add new scan types, optimize scan performance.

Pipeline Security Architecture

End-to-End DevSecOps Pipeline

DEVSECOPS PIPELINE — END-TO-END ARCHITECTURE
===============================================

Pipeline Platform: GitHub Actions (primary) + GitLab CI (legacy services) + Jenkins (batch jobs)
Security Tools: 12 integrated tools across 5 pipeline stages
Repositories Secured: 284 (198 active, 86 archived)
Pipelines with security scanning: 268 (94.4% — 16 repositories pending onboarding)
Daily pipeline executions: 1,200 (average, peak: 2,400 during release weeks)

PIPELINE STAGES & SECURITY CONTROLS:
  ┌─────────────────────┬──────────────────────────────────┬────────────────────┬──────────────────┐
  │ Stage               │ Security Controls                │ Tools              │ Gate Policy      │
  ├─────────────────────┼──────────────────────────────────┼────────────────────┼──────────────────┤
  │ PRE-COMMIT          │ Secrets detection, license       │ git-secrets,       │ Block commit     │
  │ (local developer)   │ check, basic linting             │ pre-commit,        │ on secrets       │
  │                     │                                  │ LicenseEye         │                  │
  ├─────────────────────┼──────────────────────────────────┼────────────────────┼──────────────────┤
  │ COMMIT              │ IaC scanning (Terraform,         │ Checkov, tfsec,    │ Block merge      │
  │ (PR trigger)        │ CloudFormation, K8s manifests),  │ KICS, trivy        │ on critical      │
  │                     │ SAST (static analysis),          │ config, Snyk IaC   │ IaC issues       │
  │                     │ secrets in PR diff               │                     │                  │
  ├─────────────────────┼──────────────────────────────────┼────────────────────┼──────────────────┤
  │ BUILD               │ SAST (deep scan), SCA            │ Snyk Code,         │ Block build      │
  │ (compile/test)      │ (dependency vulnerability        │ SonarQube,         │ on critical/high │
  │                     │ scan), license compliance,       │ OWASP Dependency-  │ vulnerabilities  │
  │                     │ code quality, container image    │ Check, npm audit   │ or license       │
  │                     │ build + scan                     │                    │ violations       │
  ├─────────────────────┼──────────────────────────────────┼────────────────────┼──────────────────┤
  │ PRE-DEPLOY          │ DAST (dynamic analysis),         │ OWASP ZAP,         │ Block deploy     │
  │ (staging validation)│ container runtime scan,          │ Trivy, Clair,      │ on critical      │
  │                     │ SBOM generation, compliance      │ Anchore,           │ vulnerabilities  │
  │                     │ check against policy             │ Syft (SBOM)        │ or policy fail   │
  ├─────────────────────┼──────────────────────────────────┼────────────────────┼──────────────────┤
  │ POST-DEPLOY         │ Continuous monitoring,           │ Datadog Security,  │ Alert only       │
  │ (production)        │ RASP, vulnerability re-scan,     │ Falco, Wiz,        │ (deploy already  │
  │                     │ compliance drift detection       │ Prisma Cloud       │ happened)        │
  └─────────────────────┴──────────────────────────────────┴────────────────────┴──────────────────┘

SCAN PERFORMANCE (Pipeline Impact):
  ┌────────────────────────┬──────────────────┬──────────────────┬──────────────────┐
  │ Scan Type              │ Avg Duration     │ Pipeline Overhead│ Optimization     │
  ├────────────────────────┼──────────────────┼──────────────────┼──────────────────┤
  │ Pre-commit hooks       │ < 2 seconds      │ Local (no CI)    │ Instant feedback │
  │ IaC scanning           │ 15-45 seconds    │ +20 sec          │ Parallel, cached │
  │ SAST (Snyk Code)       │ 2-5 minutes      │ +180 sec         │ Incremental scan │
  │ SCA (dependency scan)  │ 30-90 seconds    │ +60 sec          │ Diff-based scan  │
  │ Container image scan   │ 45-120 seconds   │ +90 sec          │ Layer caching    │
  │ DAST (OWASP ZAP)       │ 3-8 minutes      │ +300 sec         │ Baseline scan    │
  │ SBOM generation        │ 10-30 seconds    │ +20 sec          │ Incremental      │
  │ License compliance     │ 15-45 seconds    │ +30 sec          │ Cached results   │
  │ Secrets detection      │ 5-15 seconds     │ +10 sec          │ PR diff only     │
  └────────────────────────┴──────────────────┴──────────────────┴──────────────────┘
  Total security scan time: 5-15 minutes (average: 8 minutes)
  Total pipeline time (with security): 12-25 minutes (average: 18 minutes)
  Pipeline time without security: 8-15 minutes (average: 12 minutes)
  Security overhead: +50% pipeline time (accepted trade-off for security)

SECURITY GATE CONFIGURATION:
  ┌────────────────────────────────────┬────────────────────┬──────────────────┬────────────────────┐
  │ Gate                               │ Threshold          │ Action           │ Escalation         │
  ├────────────────────────────────────┼────────────────────┼──────────────────┼────────────────────┤
  │ SAST critical vulnerabilities      │ 0                  │ BLOCK build      │ Slack #security    │
  │ SAST high vulnerabilities          │ 0 (new only)       │ BLOCK build      │ Slack #security    │
  │ SCA critical CVEs (CVSS ≥ 9.0)    │ 0                  │ BLOCK build      │ Slack #security    │
  │ SCA high CVEs (CVSS 7.0-8.9)      │ 0 (new only)       │ BLOCK build      │ Slack + Jira ticket│
  │ License violations (GPL, AGPL)     │ 0                  │ BLOCK build      │ Dev lead review    │
  │ Container critical vulnerabilities │ 0                  │ BLOCK deploy     │ Slack #security    │
  │ IaC critical misconfigurations     │ 0                  │ BLOCK merge      │ Slack #infra       │
  │ Secrets detected in code           │ 0                  │ BLOCK commit     │ Immediate alert    │
  │ Code quality debt ratio            │ < 5%               │ WARNING          │ Track in backlog   │
  │ Test coverage                      │ > 80%              │ WARNING          │ Track in backlog   │
  │ DAST critical findings             │ 0                  │ BLOCK deploy     │ Slack #security    │
  │ SBOM completeness                  │ 100%               │ BLOCK deploy     │ Pipeline fix       │
  └────────────────────────────────────┴────────────────────┴──────────────────┴────────────────────┘

VULNERABILITY REMEDIATION SLAS:
  ┌──────────────────────┬──────────────────┬──────────────────┬──────────────────────────┐
  │ Severity             │ CVSS Range       │ SLA (hours)      │ Pipeline Impact          │
  ├──────────────────────┼──────────────────┼──────────────────┼──────────────────────────┤
  │ Critical             │ 9.0 - 10.0       │ 24 hours         │ Block all pipelines      │
  │ High                 │ 7.0 - 8.9        │ 72 hours (3 days)│ Block new introductions  │
  │ Medium               │ 4.0 - 6.9        │ 30 days          │ Warning (no block)       │
  │ Low                  │ 0.1 - 3.9        │ 90 days          │ Track only               │
  │ Informational        │ N/A              │ Next release     │ Information only         │
  └──────────────────────┴──────────────────┴──────────────────┴──────────────────────────┘

SAST (Static Application Security Testing)

SAST — STATIC APPLICATION SECURITY TESTING
============================================

SAST Platform: Snyk Code (primary) + SonarQube (secondary, code quality + security)
Languages Covered: TypeScript/JavaScript, Java, Python, Go, C#, Rust, SQL
Analysis Type: Data flow analysis + pattern matching + ML-based anomaly detection
Scan Mode: Incremental (PR diff) for speed, full scan nightly

SAST FINDINGS (Last 30 Days):
  ┌──────────────────────────┬────────────┬────────────┬────────────┬────────────┬────────────┐
  │ Vulnerability Type       │ Critical   │ High       │ Medium     │ Low        │ Total      │
  ├──────────────────────────┼────────────┼────────────┼────────────┼────────────┼────────────┤
  │ SQL Injection            │ 2          │ 4          │ 8          │ 2          │ 16         │
  │ Cross-Site Scripting     │ 0          │ 3          │ 12         │ 8          │ 23         │
  │ Hardcoded secrets        │ 3          │ 2          │ 1          │ 0          │ 6          │
  │ Path traversal           │ 1          │ 2          │ 4          │ 1          │ 8          │
  │ Insecure deserialization │ 0          │ 1          │ 3          │ 0          │ 4          │
  │ CSRF                     │ 0          │ 1          │ 5          │ 3          │ 9          │
  │ Open redirect            │ 0          │ 0          │ 6          │ 4          │ 10         │
  │ Insecure random          │ 0          │ 1          │ 2          │ 1          │ 4          │
  │ XXE                      │ 0          │ 0          │ 2          │ 0          │ 2          │
  │ Command injection        │ 1          │ 1          │ 0          │ 0          │ 2          │
  │ LDAP injection           │ 0          │ 0          │ 1          │ 0          │ 1          │
  │ Header injection         │ 0          │ 1          │ 2          │ 1          │ 4          │
  │ Mass assignment          │ 0          │ 0          │ 3          │ 2          │ 5          │
  │ Security misconfig.      │ 0          │ 2          │ 8          │ 5          │ 15         │
  │ Other                    │ 0          │ 1          │ 4          │ 3          │ 8          │
  └──────────────────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘
  Total findings: 117 | Critical: 6 | High: 18 | Medium: 61 | Low: 27

  By Language:
    TypeScript/JavaScript: 48 findings (41.0%) — XSS, open redirect dominant
    Java: 32 findings (27.4%) — SQL injection, deserialization
    Python: 18 findings (15.4%) — command injection, path traversal
    Go: 10 findings (8.5%) — CSRF, insecure random
    C#: 7 findings (6.0%) — SQL injection, header injection
    SQL: 2 findings (1.7%) — SQL injection
    Rust: 0 findings (0.0%) ✓

SAST FALSE POSITIVE MANAGEMENT:
  Total false positives identified: 23 (19.7% of findings)
    Common FP patterns:
      1. XSS in test files (test code mimics vulnerability patterns) — 8 FPs
         Resolution: Exclude test directories from XSS scan rules
      2. SQL injection in ORM queries (parameterized, flagged by pattern match) — 6 FPs
         Resolution: Tune data flow analysis to recognize ORM parameterization
      3. Hardcoded secrets in example/documentation files — 5 FPs
         Resolution: Exclude docs/, examples/, samples/ directories
      4. Path traversal in file validation logic (false alarm on safe code) — 3 FPs
         Resolution: Whitelist specific validation patterns
      5. Open redirect in OAuth callback URLs (legitimate redirect) — 1 FP
         Resolution: Whitelist OAuth provider domains

  FP reduction process:
    1. Security team reviews flagged FPs weekly
    2. Confirm FP → add to suppression list (with justification, expiry date)
    3. Suppression reviewed quarterly (remove if code changed)
    4. Tool tuning → reduce future FPs (update scan rules)
    5. Developer training → prevent FP-prone patterns

SONARQUBE CODE QUALITY + SECURITY:
  Repositories monitored: 284 (all active repositories)
  Quality Gate: "Company Standard" (enforced on merge)
  
  Quality Gate Conditions:
    New code coverage: > 80% (current avg: 84.2%)
    New security rating: A or B (no critical or high vulnerabilities)
    New reliability rating: A or B
    New maintainability rating: A or B or C
    No duplicated blocks > 3%
    No copy-paste bugs
    Smell density: < 1.7 per 100 lines
  
  Code Quality Trends (Last 6 Months):
    Avg coverage: 78.4% → 84.2% (+5.8%, improving)
    Security hotspots (unresolved): 156 → 89 (-43%, improving)
    Code smells (total): 2,340 → 1,890 (-19.2%, improving)
    Technical debt ratio: 4.2% → 2.8% (-1.4%, improving)

SCA (Software Composition Analysis)

SCA — SOFTWARE COMPOSITION ANALYSIS
=====================================

SCA Platform: Snyk Open Source (primary) + OWASP Dependency-Check (secondary) + npm audit (Node.js)
Languages/Packages: npm, pip, Maven, Go modules, NuGet, Cargo, Ruby gems
Scan Mode: Automated (every dependency change) + Scheduled (weekly full scan)

DEPENDENCY INVENTORY:
  Total unique dependencies across all repos: 4,280
    npm packages: 2,140 (50.0%)
    Python packages: 890 (20.8%)
    Java (Maven) packages: 680 (15.9%)
    Go modules: 340 (7.9%)
    NuGet packages: 120 (2.8%)
    Other: 110 (2.6%)
  
  Dependencies with known vulnerabilities: 312 (7.3%)
    Critical (CVSS ≥ 9.0): 18
    High (CVSS 7.0-8.9): 64
    Medium (CVSS 4.0-6.9): 186
    Low (CVSS 0.1-3.9): 44

TOP VULNERABLE DEPENDENCIES (by frequency of use):
  ┌─────────────────────────────┬────────────┬────────────┬────────────┬─────────────────────┐
  │ Package                     │ Vulnerable │ Repos Using │ CVSS       │ Fix Available       │
  ├─────────────────────────────┼────────────┼────────────┼────────────┼─────────────────────┤
  │ lodash < 4.17.21            │ Prototype  │ 12         │ 7.2 (High) │ Upgrade to 4.17.21  │
  │ pollyfills (various)        │ injection  │            │            │                     │
  │ axios < 1.6.0               │ ReDoS      │ 8          │ 7.5 (High) │ Upgrade to 1.6.0+   │
  │ express < 4.18.2            │ Open       │ 6          │ 6.5 (Med)  │ Upgrade to 4.18.2+  │
  │ redirect                    │            │            │            │                     │
  │ django < 4.2.7              │ SQL inj.   │ 4          │ 7.5 (High) │ Upgrade to 4.2.7+   │
  │ requests < 2.31.0           │ Info leak  │ 3          │ 5.3 (Med)  │ Upgrade to 2.31.0+  │
  │ spring-core < 6.0.12       │ RCE        │ 2          │ 9.8 (Crit) │ Upgrade to 6.0.12+  │
  │ log4j-core < 2.17.1        │ Log4Shell  │ 0          │ 10.0       │ FIXED (all upgraded)│
  │ flask < 2.3.2               │ Session    │ 2          │ 7.5 (High) │ Upgrade to 2.3.2+   │
  │ hijack                     │            │            │            │                     │
  │ golang.org/x/net < 0.17.0  │ HTTP       │ 3          │ 7.4 (High) │ Upgrade to 0.17.0+  │
  │ vulnerability              │            │            │            │                     │
  └─────────────────────────────┴────────────┴────────────┴────────────┴─────────────────────┘

DEPENDENCY FIX AUTOMATION:
  Auto-fix attempts (via Dependabot + Snyk): 156 PRs created (last 30 days)
    Merged automatically: 89 (57.1%) — minor version bumps, passing tests
    Merged after review: 42 (26.9%) — major version bumps, breaking changes
    Failed tests: 14 (9.0%) — rollback, manual investigation needed
    Superseded: 8 (5.1%) — newer fix available, old PR auto-closed
    Rejected: 3 (1.9%) — not applicable, false positive, accepted risk

LICENSE COMPLIANCE:
  Approved licenses: MIT, BSD-2, BSD-3, Apache-2.0, ISC, PostgreSQL, BlueOak-1.0, Unlicense
  Restricted licenses (require legal review): LGPL, MPL, EPL, CDDL, WTFPL
  Blocked licenses (cannot use): GPL-2.0, GPL-3.0, AGPL-3.0, SSPL, BUSL
  
  License violations found: 8 (last 30 days)
    GPL-2.0 dependency in proprietary project: 2 cases → replaced with MIT alternative
    AGPL-3.0 dependency in SaaS product: 1 case → replaced (legal risk: copyleft viral)
    LGPL dependency (dynamic link OK): 3 cases → approved by legal (dynamic linking allowed)
    Unclear license: 2 cases → flagged for developer to verify with upstream

SBOM (Software Bill of Materials):
  Format: CycloneDX 1.5 (JSON) + SPDX 2.3 (for compliance)
  Generation: Automated at build stage (Syft tool)
  Storage: Artifact registry (with each build artifact) + Central SBOM repository
  Distribution: Provided to customers upon request (compliance requirement)
  
  SBOM completeness: 98.2% (all production services have SBOMs)
  Missing SBOMs: 5 (batch jobs, legacy services — remediation planned Q2 2025)
  SBOM update frequency: Every build (production) + Weekly (staging)

Container & Secrets Security

CONTAINER SECURITY — IMAGE + RUNTIME
=======================================

Container Image Scanning: Trivy (build stage) + Snyk Container (pre-deploy stage)
Container Registry: AWS ECR (primary) + Docker Hub (public base images) + GitLab Registry (internal)
Images Scanned: 100% (all images pushed to registry are scanned before deployment)
Runtime Security: Falco (syscall monitoring) + Datadog Container Security + Prisma Cloud

CONTAINER SCAN RESULTS (Last 30 Days):
  Images scanned: 842 (new builds)
  Images with vulnerabilities: 234 (27.8%)
  Images blocked from deployment: 12 (1.4%) — critical vulnerabilities with no fix
  
  Vulnerability breakdown:
    Critical: 18 vulnerabilities (12 images blocked, 6 accepted with justification)
    High: 64 vulnerabilities (all tracked, remediation SLA: 72 hours)
    Medium: 186 vulnerabilities (tracked, remediation SLA: 30 days)
    Low: 44 vulnerabilities (informational)
  
  Base image vulnerabilities (inherited from OS layers):
    Ubuntu 22.04: 24 known CVEs (12 medium, 8 low, 4 informational) — patched in monthly image update
    Alpine 3.18: 8 known CVEs (4 medium, 4 low) — minimal footprint, preferred base
    Distroless: 0 CVEs (no package manager, no shell) — most secure, used for production
    Node 20-slim: 16 known CVEs (8 medium, 6 low, 2 informational) — updated bi-weekly

  Best practice recommendations (Trivy policy checks):
    Images not running as non-root: 18 → FIXED (added USER directive)
    Images with healthcheck missing: 34 → WARNING (not blocking, recommended)
    Images larger than 500MB: 12 → WARNING (optimize build, use multi-stage)
    Images using 'latest' tag: 4 → BLOCKED (require specific version tags)

SECRETS DETECTION:
  Tools: gitleaks (pre-commit + PR scan) + TruffleHog (deep scan, weekly) + Snyk (in-line IDE)
  Secret types detected: API keys, passwords, tokens, certificates, SSH keys, connection strings
  
  Secrets found (last 30 days):
    Pre-commit blocked: 46 (caught before commit — developers fixed immediately)
    PR scan blocked: 8 (caught in CI — developer removed secret, rotated credential)
    Deep scan found: 3 (in git history — gitleaks --log-opts, requires git filter-branch)
    Total secrets exposed: 57 (all rotated, all code cleaned)
  
  Most common secret types:
    AWS access keys: 18 (31.6%) — developers using personal AWS accounts for local testing
    GitHub tokens: 14 (24.6%) — hardcoded in scripts, should use environment variables
    Database connection strings: 10 (17.5%) — hardcoded in config files, should use Vault
    API keys (various): 9 (15.8%) — third-party service keys in code
    JWT secrets: 4 (7.0%) — hardcoded signing secrets
    SSH private keys: 2 (3.5%) — embedded in deployment scripts
  
  Secret remediation process:
    1. Detect: Pipeline fails, highlights file + line + secret type
    2. Rotate: Credential rotated immediately (assume compromised)
    3. Remove: Developer removes secret from code
    4. Replace: Use environment variable, Vault, or secret manager
    5. Verify: Re-run secret scan to confirm removal
    6. History: If secret committed before, scrub git history (git filter-branch or BFG)

Integration Points

Edge Cases