Container Image Security: Supply Chain from Build to Production
Container image security covers the full pipeline from base image choice to production admission. The four layers: secure base images (distroless, minimal), vulnerability scanning in CI (Trivy), image signing at push (Cosign + Sigstore), and admission control that enforces both (Kyverno or Policy Controller). A weak link at any layer lets vulnerable or unsigned images reach production.
The container image supply chain has four distinct attack surfaces: the base image (vulnerabilities from the OS packages), the build pipeline (compromised dependencies pulled at build time), the registry (tampered images after push), and admission (no gate preventing vulnerable images from running). Most teams focus on scanning and ignore signing; most incidents come from the layer they didn't address.
This is the full pipeline with tooling at each layer.
Layer 1: Base Image Selection
Distroless (from Google) strips everything from the container except the runtime — no shell, no package manager, no debug tools. The attack surface is minimal because there's nothing to exploit:
1# Multi-stage: build in full image, run in distroless
2FROM golang:1.23-alpine AS builder
3WORKDIR /app
4COPY go.mod go.sum ./
5RUN go mod download
6COPY . .
7RUN CGO_ENABLED=0 go build -o payments-api ./cmd/server
8
9FROM gcr.io/distroless/static-debian12:nonroot AS runtime
10# nonroot variant: runs as UID 65532 (nonroot user) by default
11COPY --from=builder /app/payments-api /payments-api
12EXPOSE 8080
13ENTRYPOINT ["/payments-api"]Distroless variants:
gcr.io/distroless/static-debian12— static binaries only (Go, Rust)gcr.io/distroless/base-debian12— glibc for C/C++ binariesgcr.io/distroless/cc-debian12— C++ standard librarygcr.io/distroless/python3-debian12— Python runtime onlygcr.io/distroless/java21-debian12— JRE only
For interpreted languages (Node.js, Python), use Alpine or Debian slim if distroless doesn't work with your dependencies:
1FROM python:3.13-slim AS builder
2# ... build steps
3
4FROM python:3.13-slim AS runtime
5# Combine multi-stage to avoid dev tools in final image
6COPY --from=builder /app /app
7RUN groupadd -r appgroup && useradd -r -g appgroup appuser
8USER appuser
9ENTRYPOINT ["python", "-m", "app"]Layer 2: Trivy Vulnerability Scanning in CI
Trivy (Aqua Security) scans container images, filesystems, Git repos, and Kubernetes manifests for vulnerabilities, exposed secrets, and misconfigurations.
GitHub Actions Integration
1# .github/workflows/ci.yml
2name: Build and Scan
3
4on:
5 push:
6 branches: [main]
7 pull_request:
8
9jobs:
10 build-and-scan:
11 runs-on: ubuntu-latest
12
13 steps:
14 - uses: actions/checkout@v4
15
16 - name: Build image
17 run: docker build -t payments-api:${{ github.sha }} .
18
19 - name: Run Trivy vulnerability scan
20 uses: aquasecurity/trivy-action@master
21 with:
22 image-ref: payments-api:${{ github.sha }}
23 format: sarif
24 output: trivy-results.sarif
25 severity: CRITICAL,HIGH
26 exit-code: '1' # Fail the pipeline on CRITICAL/HIGH
27 ignore-unfixed: true # Skip vulnerabilities with no fix yet
28
29 - name: Upload Trivy results to GitHub Security
30 uses: github/codeql-action/upload-sarif@v3
31 if: always() # Upload even if scan fails
32 with:
33 sarif_file: trivy-results.sarif
34
35 - name: Scan IaC and Kubernetes manifests
36 uses: aquasecurity/trivy-action@master
37 with:
38 scan-type: config
39 scan-ref: k8s/
40 exit-code: '1'
41 severity: HIGH,CRITICALTrivy CLI for Local Use
1# Install
2curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | \
3 sh -s -- -b /usr/local/bin v0.58.0
4
5# Scan an image (pulls from registry if not local)
6trivy image payments-api:latest
7
8# Scan with only HIGH/CRITICAL (for CI enforcement)
9trivy image --severity HIGH,CRITICAL payments-api:latest
10
11# Ignore specific CVEs (create a .trivyignore file with CVE IDs, one per line)
12# CVE-2024-12345
13trivy image --ignorefile .trivyignore payments-api:latest
14
15# Scan Kubernetes manifests
16trivy config ./k8s/
17
18# Generate SBOM (Software Bill of Materials)
19trivy image --format cyclonedx --output sbom.json payments-api:latest
20
21# Scan a running cluster
22trivy k8s --report summary clusterTrivy in ECR (Private Registry Scan)
# Authenticate Trivy to ECR
aws ecr get-login-password | docker login \
--username AWS \
--password-stdin 123456789.dkr.ecr.us-east-1.amazonaws.com
trivy image 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:latestLayer 3: Cosign Image Signing
Cosign (part of Sigstore) signs container images so you can verify at admission that an image came from your trusted CI pipeline and hasn't been modified.
Keyless Signing with Sigstore (Recommended)
Keyless signing uses short-lived certificates from Sigstore's Fulcio CA tied to your CI/CD identity (GitHub Actions OIDC, Google Workload Identity, etc.) — no private key to manage:
1# .github/workflows/sign.yml
2- name: Install Cosign
3 uses: sigstore/cosign-installer@v3.7.0
4
5- name: Log in to ECR
6 run: |
7 aws ecr get-login-password | docker login \
8 --username AWS --password-stdin \
9 123456789.dkr.ecr.us-east-1.amazonaws.com
10
11- name: Build and push image
12 run: |
13 docker build -t 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:${{ github.sha }} .
14 docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:${{ github.sha }}
15
16- name: Sign image (keyless — uses GitHub OIDC)
17 run: |
18 cosign sign --yes \
19 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:${{ github.sha }}The signature is stored in the same registry as a tag: sha256-<digest>.sig. No separate signature storage needed.
Verifying Signatures
1# Verify with keyless (checks Rekor transparency log)
2# COSIGN_EXPERIMENTAL is not needed in Cosign v2.x — keyless is the default
3cosign verify \
4 --certificate-identity-regexp "https://github.com/my-org/payments-api/.github/workflows/.*" \
5 --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
6 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:latest
7
8# Successful output:
9# Verification for 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:latest --
10# The following checks were performed on each of these signatures:
11# - The cosign claims were validated
12# - Existence of the claims in the transparency log was verified online
13# - The code signing certificate claims were validatedAttach SBOM to Image
# Attest the SBOM to the image (cosign v2.x — "attach sbom" was removed in v2.0)
cosign attest \
--predicate sbom.json \
--type cyclonedx \
--yes \
123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:${{ github.sha }}Verifying Build Integrity: SLSA Provenance (2026 Standard)
In 2026, signing isn't enough; we need to prove how an image was built. SLSA (Supply-chain Levels for Software Artifacts) provides non-forgeable provenance that links an image back to its source code and build workflow:
1# Use the SLSA GitHub Generator to build and sign
2jobs:
3 build:
4 permissions:
5 id-token: write
6 contents: read
7 actions: read
8 uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
9 with:
10 image: 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api
11 digest: ${{ needs.build.outputs.digest }}This ensures that the image in your registry was built by your official workflow and not a compromised local environment. Kyverno can then enforce that only images with valid SLSA Level 3 provenance are allowed to run in production.
Layer 4: Admission Enforcement with Kyverno
Kyverno policies at admission time enforce that only scanned and signed images reach production:
Require Image Digest (Not Tag)
Tags are mutable — the same tag can point to different images after a push. Digests are immutable:
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: require-image-digest
5spec:
6 validationFailureAction: Enforce
7 background: false
8
9 rules:
10 - name: require-digest
11 match:
12 any:
13 - resources:
14 kinds: [Pod]
15 namespaces: [production, staging]
16
17 validate:
18 message: "Images must use a digest (@sha256:...) not a mutable tag."
19 foreach:
20 - list: "request.object.spec.containers"
21 deny:
22 conditions:
23 any:
24 - key: "{{ element.image }}"
25 operator: NotContains
26 value: "@sha256:"Require Signed Images (Cosign Keyless)
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: require-signed-images
5spec:
6 validationFailureAction: Enforce
7 background: false
8
9 rules:
10 - name: verify-image-signature
11 match:
12 any:
13 - resources:
14 kinds: [Pod]
15 namespaces: [production]
16
17 verifyImages:
18 - imageReferences:
19 - "123456789.dkr.ecr.us-east-1.amazonaws.com/*"
20
21 attestors:
22 - count: 1
23 entries:
24 - keyless:
25 subject: "https://github.com/my-org/*"
26 issuer: "https://token.actions.githubusercontent.com"
27 rekor:
28 url: https://rekor.sigstore.dev
29
30 mutateDigest: true # Rewrite tag → digest after verification
31 verifyDigest: true # Also verify the digest is correct
32 required: true # Fail if no valid signature foundBlock Images with Critical Vulnerabilities (via Attestation)
Trivy can attest scan results; Kyverno can enforce that only images with a passing attestation run:
1# Generate Trivy scan results in Cosign's vulnerability attestation format (not SARIF)
2# This produces JSON compatible with predicateType: https://cosign.sigstore.dev/attestation/vuln/v1
3trivy image --format cosign-vuln --output trivy-vuln.json payments-api:latest
4
5cosign attest \
6 --predicate trivy-vuln.json \
7 --type https://cosign.sigstore.dev/attestation/vuln/v1 \
8 --yes \
9 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api:${{ github.sha }}
10# Note: --format sarif produces SARIF output, not the Cosign vuln attestation schema.
11# Use --format cosign-vuln to get a predicate compatible with predicateType above.1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: require-vuln-scan-attestation
5spec:
6 validationFailureAction: Enforce
7
8 rules:
9 - name: verify-trivy-attestation
10 match:
11 any:
12 - resources:
13 kinds: [Pod]
14 namespaces: [production]
15
16 verifyImages:
17 - imageReferences:
18 - "123456789.dkr.ecr.us-east-1.amazonaws.com/*"
19
20 attestations:
21 - predicateType: https://cosign.sigstore.dev/attestation/vuln/v1
22 attestors:
23 - entries:
24 - keyless:
25 subject: "https://github.com/my-org/*"
26 issuer: "https://token.actions.githubusercontent.com"Blocking Latest Tag
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: block-latest-tag
5spec:
6 validationFailureAction: Enforce
7 background: false
8
9 rules:
10 - name: no-latest
11 match:
12 any:
13 - resources:
14 kinds: [Pod]
15 namespaces: [production, staging]
16
17 validate:
18 message: "Use a specific image tag or digest. 'latest' is not allowed in production."
19 foreach:
20 - list: "request.object.spec.containers"
21 deny:
22 conditions:
23 any:
24 - key: "{{ element.image }}"
25 operator: EndsWith
26 value: ":latest"
27 - key: "{{ element.image }}"
28 operator: Contains
29 value: ":latest@"Frequently Asked Questions
Is distroless compatible with kubectl exec for debugging?
No — there's no shell to exec into. This is intentional. For debugging, use kubectl debug with an ephemeral container that has a shell. See Kubernetes Debugging and Troubleshooting Guide. In production, the lack of a shell is the security property — a compromised container that can't exec a shell significantly limits lateral movement.
Does Trivy scanning need to run on every PR or just main?
Both, for different reasons. On PRs: catch vulnerabilities before they merge (cheaper to fix). On main/push: scan the actual built image digest that will be deployed (the build may pull different dependency versions than what was on the PR). ECR also offers on-push scanning — enable it as a second layer, not a replacement for CI scanning.
What happens if Cosign is down during deployment?
Kyverno's verifyImages calls Sigstore's Rekor (transparency log) and Fulcio for verification. If these services are unavailable, verification will fail and pods won't start. For highly regulated environments, run your own private Sigstore stack (Rekor + Fulcio + TUF). For keyless verification, Kyverno caches verification results in a configurable store to handle transient outages.
For Kyverno policies that enforce these image requirements alongside other security controls, see Kubernetes Admission Webhooks: OPA Gatekeeper and Kyverno. For CIS benchmark hardening that complements supply chain security with runtime controls, see Kubernetes Security Hardening and CIS Benchmarks.
Setting up a supply chain security pipeline for a Kubernetes platform? Talk to us at Coding Protocols — we help platform teams implement image signing, scanning, and admission enforcement that keeps the build-to-production pipeline secure without slowing down developer velocity.
