OutputValues icon
Labels icon
Commands icon
Docker icon
If icon
Log icon
Exit icon
Pause icon
Assert icon
Webhook icon

Patch container images with Copacetic on registry push, after approval and a Trivy re-scan

Scan each pushed image with Trivy, approve the fix, patch it with Copacetic and BuildKit, re-scan, and publish the patched tag only when the re-scan improved.

Categories
Infrastructure

Fixing the vulnerabilities a scanner finds in an image usually means waiting for whoever owns the Dockerfile to rebuild it. Copacetic (copa, a CNCF project) patches the existing image instead, upgrading only the vulnerable OS packages in one new layer. This flow runs it on every push to your registry: Trivy scans the image, a person approves the list of fixes, Copacetic patches it through BuildKit, Trivy scans the result, and <tag>-patched is published next to the original only if the re-scan found fewer fixable vulnerabilities.

This blueprint was created by kruxarth.

How it differs from the Copacetic hotpatch engine blueprint

copacetic-container-vulnerability-hotpatch-engine runs every 6 hours for one image set in its inputs, and decides from severity counts read from a scanner metrics endpoint. This flow:

  • Starts from the registry. The push notification is the trigger, so each pushed image gets its own execution within seconds. There is no schedule and no list of images to keep up to date.
  • Patches what it scanned. Trivy scans the pushed digest, Copacetic gets that Trivy report, and copa patch runs on the digest rather than the tag.
  • Publishes only after a re-scan. The patched image goes to a staging repository first. Trivy scans it again, and <tag>-patched is published only when the re-scan finds a new digest with fewer fixable vulnerabilities.

How it works

  1. on_push (io.kestra.plugin.core.trigger.Webhook) receives the registry's push notification. Only a tagged single-platform manifest starts an execution; the flow's own pushes get a 204 and start nothing. You can also execute it by hand with a repository and tag.
  2. scan_source (io.kestra.plugin.scripts.shell.Commands in the Trivy image) scans the image against a Trivy server, keeping OS package vulnerabilities that have a fixed version. before counts them by severity and package.
  3. Nothing fixable: the execution ends with outcome: nothing-to-patch.
  4. Otherwise show_findings logs each vulnerability with its installed and fixed version, and approve (Pause) waits for PATCH or REJECT and a note. The form shows the counts. Without an answer within approval_window (7 days), the execution is cancelled.
  5. REJECT ends in WARNING with outcome: rejected. Nothing is built or pushed.
  6. PATCH: patch runs copa patch on the scanned digest. BuildKit upgrades the reported packages and pushes the result to copa-staging/<repository>:<tag>-patched, with an OpenVEX document of the fixes.
  7. scan_patched scans the staged image, and verify (Assert) fails the execution unless it is a new digest with fewer fixable vulnerabilities.
  8. publish copies that digest to <repository>:<tag>-patched with regctl and checks the tag resolves to it. The original tag is never touched.

What you get

  • The fix is applied, not suggested, and only after someone approved it with a note.
  • The published image is byte for byte the one that passed the re-scan; a patch that did not help is never published.
  • In a test, nginx:1.29.0-alpine went from 221 fixable vulnerabilities (147 CVEs, 3 critical) to 0: Copacetic upgraded 20 packages in 9 seconds, the tag was published 11 seconds after approval, and nginx -t still passed in the patched image.
  • Both Trivy reports and the OpenVEX file stay with the execution, and the labels outcome, fixable_before and fixable_after make every image's history searchable.

Who it's for

  • Platform and security teams that run a registry and use images they don't build: vendor images, base images, internal images whose owners have moved on.
  • Teams that need a fix today and a record of the decision, while the proper rebuild goes through the normal pipeline.

Why orchestrate this with Kestra

  • The registry calls Kestra directly, so a push is scanned within seconds without a poller.
  • Pause with onResume inputs gives an approval form in the UI and an API endpoint, and each image is its own execution with logs, labels and outputs.
  • Each tool runs in its own pinned container through the Docker task runner, and the whole process is one readable YAML file instead of a CI script.

Prerequisites

  • Docker on the Kestra host, with its socket available to Kestra. The task containers use the host network, so localhost is the Docker host.

  • A registry that sends push notifications. With Distribution (registry:3), add this to its config.yml:

    notifications:
      endpoints:
        - name: kestra
          url: http://localhost:8080/api/v1/main/executions/webhook/company.security/copacetic-patch-on-registry-push/<REGISTRY_WEBHOOK_KEY>
          timeout: 5s
          threshold: 3
          backoff: 10s
          ignore:
            mediatypes: [application/octet-stream]
            actions: [pull]
    
    docker run -d --name registry --network host -v $PWD/config.yml:/etc/distribution/config.yml:ro registry:3.1.2
    
  • A BuildKit daemon told that the registry speaks plain HTTP, and a Trivy server, which keeps the vulnerability database:

    printf '[registry."localhost:5000"]\n  http = true\n' > buildkitd.toml
    docker run -d --name buildkitd --network host --privileged -v $PWD/buildkitd.toml:/etc/buildkit/buildkitd.toml:ro moby/buildkit:v0.34.1 --addr tcp://127.0.0.1:1234
    docker run -d --name trivy-server --network host -v trivy-cache:/root/.cache/trivy aquasec/trivy:0.76.0 server --listen 127.0.0.1:4954
    
  • Network access from BuildKit to the distribution's package mirrors (for example dl-cdn.alpinelinux.org).

Secrets

  • REGISTRY_WEBHOOK_KEY: the key in the notification URL. Anyone who knows it can start a scan.

Quick start

  1. Start the registry, BuildKit and the Trivy server, add the secret, and save the flow.
  2. Push an old image: docker pull nginx:1.29.0-alpine && docker tag nginx:1.29.0-alpine localhost:5000/demo/nginx:1.29.0-alpine && docker push localhost:5000/demo/nginx:1.29.0-alpine.
  3. An execution starts and pauses. Read the list in its logs, click Resume, choose PATCH and add a note.
  4. When it succeeds, localhost:5000/demo/nginx:1.29.0-alpine-patched exists and the flow outputs give its digest and both counts. The original tag still points to the original image.

Things to know

  • Copacetic patches OS packages (apk, apt, yum/dnf). Application dependencies are left out of the scans and still need a rebuild.
  • A multi-platform index does not start an execution (its per-platform manifests are pushed untagged). Push single-platform tags, or extend the flow with Copacetic's --platform.
  • Kestra keeps the decision, the note and when it was made. Put the approver or the change ticket in the note.
  • Flow outputs are set when an image is published; for the other outcomes, read the labels.
  • Running the flow again for a tag replaces <tag>-patched. The staging repository keeps the last candidate for when verify fails.
  • For a registry with authentication or HTTPS, give the three tool containers credentials (for example a mounted Docker config.json) and set registry_tls: enabled. BuildKit and the Trivy server run here without TLS or a token, so keep them on localhost.

How to extend

  • Notify the approvers from onPause on approve, with a link to the execution.
  • Sign the published digest with cosign, or roll it out to a Kubernetes deployment, after publish.
  • Add a nightly Schedule that runs the flow for the tags you deploy, so images are re-checked when new CVEs are published, not only on push.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.