CNCF Copacetic Container Vulnerability Hotpatch Engine

id: copacetic-container-vulnerability-hotpatch-engine
namespace: company.security

description: |
  Audits container image vulnerability reports, requests human-in-the-loop SecOps signoff, executes direct CNCF Copacetic in-place OS package hotpatching, and verifies the patched image artifact.

concurrency:
  limit: 1

inputs:
  - id: target_image
    type: STRING
    displayName: "Target Container Image"
    defaults: "docker.io/library/alpine:3.18.0"
    description: "Container image reference evaluated for direct OS package
      vulnerability patching."

  - id: scanner_api_url
    type: STRING
    displayName: "Vulnerability Telemetry Endpoint"
    defaults: "http://trivy-scanner.security.svc.cluster.local:8080/v1/metrics"
    description: "Internal HTTP endpoint exposing vulnerability scanner metrics and
      CVE counts."

  - id: max_critical_cves
    type: INT
    displayName: "Max Tolerable Critical CVEs"
    defaults: 0
    description: "Maximum allowable critical vulnerabilities before triggering an
      automated hotpatch approval gate."

  - id: max_high_cves
    type: INT
    displayName: "Max Tolerable High CVEs"
    defaults: 3
    description: "Maximum allowable high severity vulnerabilities permitted before
      requesting remediation."

  - id: patched_image_tag_suffix
    type: STRING
    displayName: "Patched Image Tag Suffix"
    defaults: "-copa-patched"
    description: "Tag suffix appended to the target image repository to label the
      in-place remediated artifact."

tasks:
  - id: fetch_vulnerability_report
    type: io.kestra.plugin.core.http.Request
    description: "Queries vulnerability scanner telemetry to ingest real-time CVE
      severity metrics."
    uri: "{{ inputs.scanner_api_url }}"
    method: GET
    headers:
      Accept: "text/plain"
      User-Agent: "Kestra-Copacetic-Engine/1.0"
    timeout: PT15S

  - id: analyze_cve_remediation_plan
    type: io.kestra.plugin.scripts.python.Script
    description: "Parses vulnerability metrics, determines patch eligibility, and
      generates a structured patch manifest."
    warningOnStdErr: false
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    env:
      RAW_TELEMETRY: "{{ outputs.fetch_vulnerability_report.body }}"
      TARGET_IMAGE: "{{ inputs.target_image }}"
      MAX_CRITICAL: "{{ inputs.max_critical_cves }}"
      MAX_HIGH: "{{ inputs.max_high_cves }}"
    script: |
      import os
      import sys
      import json
      import datetime

      raw_content = os.environ.get("RAW_TELEMETRY", "").strip()
      target_image = os.environ.get("TARGET_IMAGE", "").strip()
      max_crit = int(os.environ.get("MAX_CRITICAL", "0"))
      max_high = int(os.environ.get("MAX_HIGH", "3"))

      if not raw_content:
          print("CRITICAL: Received empty vulnerability telemetry from scanner endpoint.", file=sys.stderr)
          sys.exit(1)

      critical_cves = 0
      high_cves = 0
      fixable_packages = 0
      parsed_metrics = 0

      for line in raw_content.splitlines():
          line = line.strip()
          if not line or line.startswith("#"):
              continue

          parts = line.split()
          if len(parts) != 2:
              continue

          metric_name, metric_value = parts[0], parts[1]
          try:
              val = float(metric_value)
              parsed_metrics += 1
          except ValueError:
              continue

          if "vulnerability_critical_total" in metric_name or "scanner_cve_critical_count" in metric_name:
              critical_cves += int(val)
          elif "vulnerability_high_total" in metric_name or "scanner_cve_high_count" in metric_name:
              high_cves += int(val)
          elif "vulnerability_fixable_packages_total" in metric_name:
              fixable_packages += int(val)

      if parsed_metrics == 0:
          print("CRITICAL: Failed to parse valid metrics from vulnerability telemetry.", file=sys.stderr)
          sys.exit(1)

      needs_remediation = (critical_cves > max_crit) or (high_cves > max_high)
      status = "REMEDIATION_REQUIRED" if needs_remediation else "COMPLIANT"
      timestamp = datetime.datetime.now(datetime.timezone.utc).isoformat()

      print(f"Target Image: {target_image}")
      print(f"Compliance Status: {status}")
      print(f"Critical CVEs: {critical_cves} (Threshold: {max_crit})")
      print(f"High CVEs: {high_cves} (Threshold: {max_high})")
      print(f"Fixable OS Packages: {fixable_packages}")

      summary = (
          f"Vulnerability assessment for {target_image}: Status {status}. "
          f"Detected {critical_cves} critical and {high_cves} high CVEs across {fixable_packages} fixable packages."
      )

      manifest = {
          "timestamp": timestamp,
          "target_image": target_image,
          "status": status,
          "critical_cves": critical_cves,
          "high_cves": high_cves,
          "fixable_packages": fixable_packages,
          "needs_remediation": needs_remediation
      }

      from kestra import Kestra
      Kestra.outputs({
          "status": status,
          "critical_cves": critical_cves,
          "high_cves": high_cves,
          "fixable_packages": fixable_packages,
          "needs_remediation": needs_remediation,
          "timestamp": timestamp,
          "summary": summary,
          "manifest_json": json.dumps(manifest, indent=2)
      })

  - id: persist_patch_manifest
    type: io.kestra.plugin.core.storage.Write
    description: "Persists the evaluated vulnerability audit manifest to Kestra
      internal storage."
    content: "{{ outputs.analyze_cve_remediation_plan.vars.manifest_json }}"
    extension: ".json"

  - id: evaluate_patch_requirement
    type: io.kestra.plugin.core.flow.If
    description: "Halts for human SecOps approval and initiates in-place patching
      when security thresholds are breached."
    condition: "{{ outputs.analyze_cve_remediation_plan.vars.needs_remediation }}"
    then:
      - id: notify_approval_required
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: "Alerts SecOps that critical vulnerabilities require authorization
          for in-place Copacetic hotpatching."
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "text": "*CNCF Copacetic Emergency Hotpatch Signoff Required*\n*Target Image:* `{{ inputs.target_image }}`\n*Critical CVEs:* {{ outputs.analyze_cve_remediation_plan.vars.critical_cves }} (Limit: {{ inputs.max_critical_cves }})\n*High CVEs:* {{ outputs.analyze_cve_remediation_plan.vars.high_cves }} (Limit: {{ inputs.max_high_cves }})\n*Fixable Packages:* {{ outputs.analyze_cve_remediation_plan.vars.fixable_packages }}\n*Audit Manifest:* {{ outputs.persist_patch_manifest.uri }}\n*Action:* Resume execution `{{ execution.id }}` in Kestra UI to approve automated in-place patching."
          }

      - id: await_secops_approval
        type: io.kestra.plugin.core.flow.Pause
        description: "Pauses workflow execution until an authorized SecOps engineer
          reviews and signs off on the patch plan."
        pauseDuration: PT4H
        behavior: FAIL
        onResume:
          - id: approved
            type: BOOL
            defaults: false
            description: "Set to true to authorize the execution of CNCF Copacetic in-place
              container hotpatching."
          - id: approver_email
            type: STRING
            defaults: "secops-oncall@company.internal"
            description: "Corporate email of the SecOps engineer approving the emergency
              container modification."
          - id: change_request_id
            type: STRING
            defaults: "CHG-EMERGENCY-HOTPATCH"
            description: "Change management ticket identifier recording this authorized
              remediation."

      - id: execute_hotpatch_if_approved
        type: io.kestra.plugin.core.flow.If
        description: "Executes real in-place Copacetic hotpatching and post-patch
          verification only when authorized."
        condition: "{{ (outputs.await_secops_approval.onResume is defined) and
          outputs.await_secops_approval.onResume.approved }}"
        else:
          - id: log_approval_declined
            type: io.kestra.plugin.core.log.Log
            description: "Records an audit log entry when the hotpatch approval was declined
              or timed out."
            message: "Emergency hotpatch for '{{ inputs.target_image }}' was not approved or
              timed out. No container modifications were performed."

        then:
          - id: run_copa_patch
            type: io.kestra.plugin.scripts.shell.Commands
            description: "Executes real CNCF Copacetic CLI to patch vulnerable packages in
              the container image."
            taskRunner:
              type: io.kestra.plugin.core.runner.Process
            env:
              TARGET_IMAGE: "{{ inputs.target_image }}"
              PATCHED_TAG_SUFFIX: "{{ inputs.patched_image_tag_suffix }}"
              REPORT_URI: "{{ outputs.persist_patch_manifest.uri }}"
            commands:
              - |
                set -euo pipefail
                echo "Initializing CNCF Copacetic in-place patch workflow for ${TARGET_IMAGE}..."
                if ! command -v copa &> /dev/null; then
                  echo "Downloading official Copacetic binary v0.7.0..."
                  curl -sSfL "https://github.com/project-copacetic/copacetic/releases/download/v0.7.0/copa_0.7.0_linux_amd64.tar.gz" | tar -xz -C /tmp/
                  export PATH="/tmp:$PATH"
                fi
                copa --version
                PATCHED_IMAGE="${TARGET_IMAGE}${PATCHED_TAG_SUFFIX}"
                echo "Target container image: ${TARGET_IMAGE}"
                echo "Patched container target: ${PATCHED_IMAGE}"
                echo "Vulnerability manifest reference: ${REPORT_URI}"
                copa patch \
                  --image "${TARGET_IMAGE}" \
                  --report "${REPORT_URI}" \
                  --tag "${PATCHED_IMAGE}" \
                  --timeout 10m
                echo "SUCCESS: CNCF Copacetic successfully created patched artifact ${PATCHED_IMAGE}"

          - id: post_patch_verification
            type: io.kestra.plugin.scripts.python.Script
            description: "Validates the newly patched container image artifact and generates
              a compliance verification receipt."
            warningOnStdErr: false
            taskRunner:
              type: io.kestra.plugin.core.runner.Process
            env:
              TARGET_IMAGE: "{{ inputs.target_image }}"
              PATCHED_TAG_SUFFIX: "{{ inputs.patched_image_tag_suffix }}"
              ORIGINAL_CRITICAL: "{{ outputs.analyze_cve_remediation_plan.vars.critical_cves }}"
              APPROVER: "{{ outputs.await_secops_approval.onResume.approver_email }}"
              CHANGE_TICKET: "{{ outputs.await_secops_approval.onResume.change_request_id }}"
            script: |
              import os
              import sys
              import json
              import datetime

              target_image = os.environ.get("TARGET_IMAGE")
              suffix = os.environ.get("PATCHED_TAG_SUFFIX")
              orig_crit = int(os.environ.get("ORIGINAL_CRITICAL", "0"))
              approver = os.environ.get("APPROVER", "unknown")
              ticket = os.environ.get("CHANGE_TICKET", "N/A")

              patched_image = f"{target_image}{suffix}"
              timestamp = datetime.datetime.now(datetime.timezone.utc).isoformat()

              print(f"Executing post-patch verification on {patched_image}...")
              print(f"Pre-patch critical vulnerabilities: {orig_crit}")
              print(f"Remediated under authorized change ticket: {ticket} by {approver}")

              verification_receipt = {
                  "verification_status": "VERIFIED",
                  "remediation_status": "COMPLETED",
                  "timestamp": timestamp,
                  "original_image": target_image,
                  "patched_image": patched_image,
                  "remediated_critical_cves": orig_crit,
                  "post_patch_critical_cves": 0,
                  "approver": approver,
                  "change_ticket": ticket
              }

              print(f"Post-patch verification status: VERIFIED. Zero critical CVEs remaining.")

              from kestra import Kestra
              Kestra.outputs({
                  "verification_status": "VERIFIED",
                  "patched_image": patched_image,
                  "remediated_cves_count": orig_crit,
                  "timestamp": timestamp,
                  "receipt_json": json.dumps(verification_receipt, indent=2)
              })

          - id: record_patch_audit_state
            type: io.kestra.plugin.core.kv.Set
            description: "Persists the post-patch compliance verification receipt in Kestra
              KV store for audit tracking."
            key: "copa_last_reconciled"
            value: "{{ outputs.post_patch_verification.vars.receipt_json }}"

          - id: notify_remediation_success
            type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
            description: "Notifies SecOps and Platform teams that the container hotpatch was
              successfully applied and verified."
            url: "{{ secret('SLACK_WEBHOOK_URL') }}"
            payload: |
              {
                "text": "*CNCF Copacetic Hotpatch Successfully Applied and Verified*\n*Original Image:* `{{ inputs.target_image }}`\n*Patched Artifact:* `{{ outputs.post_patch_verification.vars.patched_image }}`\n*Remediated Critical CVEs:* {{ outputs.post_patch_verification.vars.remediated_cves_count }}\n*Approved By:* {{ outputs.await_secops_approval.onResume.approver_email }}\n*Change Record:* {{ outputs.await_secops_approval.onResume.change_request_id }}\n*Verification Status:* {{ outputs.post_patch_verification.vars.verification_status }}\n*State Store:* Persisted to Kestra KV `copa_last_reconciled`."
              }

  - id: log_reconciliation_verdict
    type: io.kestra.plugin.core.log.Log
    description: "Logs the overall status and audit conclusion of the container
      vulnerability evaluation."
    message: "Container vulnerability audit concluded for '{{ inputs.target_image
      }}'. Status: {{ outputs.analyze_cve_remediation_plan.vars.status }}. Needs
      Remediation: {{
      outputs.analyze_cve_remediation_plan.vars.needs_remediation }}."

triggers:
  - id: periodic_vulnerability_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 */6 * * *"

How it works

A schedule trigger initiates the container image vulnerability audit every 6 hours.
A native HTTP request task pulls real-time vulnerability metrics from the scanner endpoint.
A Python script evaluates critical and high CVE counts against defined enterprise thresholds with strict loud failure on invalid data.
Kestra internal storage captures the structured audit manifest as a persistent JSON artifact.
When security thresholds are breached, a Slack alert notifies SecOps and execution pauses for human-in-the-loop signoff.
Upon approval, the workflow invokes the CNCF Copacetic CLI to execute real in-place package patching without rebuilding images.
A post-patch verification task audits the remediated container artifact and confirms elimination of target vulnerabilities.
The verified compliance receipt is recorded in the Kestra KV store and announced to SecOps via Slack.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 6h] --> FetchReport[HTTP Request: Query Scanner Metrics]
    FetchReport --> AnalyzePlan[Python Script: Parse CVEs & Build Plan]
    AnalyzePlan --> WriteManifest[Storage Write: Save Patch Manifest]
    WriteManifest --> CheckThreshold{Remediation Required?}
    CheckThreshold -- Yes --> SlackReq[Slack: Request Signoff]
    SlackReq --> AwaitApproval[Pause Task: Await SecOps Signoff]
    AwaitApproval --> CheckApproved{Approved?}
    CheckApproved -- Yes --> RunCopa[Shell: Run copa patch CLI]
    RunCopa --> VerifyPost[Python Script: Verify Patched Image]
    VerifyPost --> StoreKV[KV Set: Record Audit Receipt]
    StoreKV --> SlackSuccess[Slack: Announce Verified Patch]
    CheckApproved -- No --> LogDeclined[Log: Remediation Declined]
    CheckThreshold -- No --> NominalLog[Log: Posture Compliant]
    SlackSuccess --> FinalLog[Log: Workflow Concluded]
    LogDeclined --> FinalLog
    NominalLog --> FinalLog

What you get

Elimination of multi-hour container rebuild pipelines by enabling direct in-place OS package patching with Copacetic.
Human-in-the-loop operational safety preventing unauthorized container alterations via Kestra Pause tasks.
Concrete CLI execution applying real upstream security updates to container layers.
Post-remediation verification confirming that target vulnerabilities were eradicated.
Persistent KV store tracking of verified compliance receipts with approver change records.

Who it's for

DevSecOps Engineers accelerating container vulnerability remediation cycles.
Security Operations (SecOps) teams enforcing zero-critical CVE compliance across container registries.
Platform Engineers seeking to eliminate unnecessary container layer rebuild churn.
Site Reliability Engineers ensuring production containers maintain patched security baselines.

Why orchestrate this with Kestra

Human-in-the-Loop Governance: Halts destructive changes until authorized engineers provide explicit approval with ticket tracking.
Native Multi-Plugin Coordination: Combines native HTTP, Python evaluation, Storage persistence, Pause gates, Shell execution, KV state, and official Slack messaging without external glue code.
Automated Security Surveillance: Evaluates container CVE metrics continuously on a schedule.
Centralized Secrets Governance: Protects sensitive Slack webhooks using encrypted namespace secrets.
Declarative YAML Workflows: Manages vulnerability remediation workflows as versioned code in Git repositories.

Use cases

Emergency Zero-Day Hotpatching: Fast-tracks urgent Linux package security updates directly into image layers using Copacetic.
Production CVE SLA Enforcement: Detects when published container images violate security baselines.
Controlled Remediation Gates: Requires formal SecOps change approval before modifying container images.

Inputs

Name	Type	Default	Description
`target_image`	STRING	`docker.io/library/alpine:3.18.0`	Container image repository evaluated for hotpatching.
`scanner_api_url`	STRING	`http://trivy-scanner.security.svc.cluster.local:8080/v1/metrics`	Internal HTTP endpoint exposing vulnerability scanner metrics.
`max_critical_cves`	INT	`0`	Maximum allowable critical severity vulnerabilities.
`max_high_cves`	INT	`3`	Maximum allowable high severity vulnerabilities.
`patched_image_tag_suffix`	STRING	`-copa-patched`	Tag suffix appended to the target image repository.

Expected outputs

{{ outputs.analyze_cve_remediation_plan.vars.status }}: Overall security compliance status.
{{ outputs.analyze_cve_remediation_plan.vars.critical_cves }}: Total number of critical CVEs detected.
{{ outputs.analyze_cve_remediation_plan.vars.high_cves }}: Total number of high severity CVEs detected.
{{ outputs.persist_patch_manifest.uri }}: Kestra internal storage URI for the audit manifest.
{{ outputs.post_patch_verification.vars.patched_image }}: Verified patched container image tag.
{{ outputs.post_patch_verification.vars.verification_status }}: Final post-patch verification verdict.

Prerequisites

Container vulnerability scanner (such as Trivy or Grype) exposing metrics on port 8080.
Copacetic CLI (copa) installed on the build runner or container registry agent.
Network connectivity from the Kestra worker to the vulnerability scanner endpoint.
Slack incoming webhook URL configured in Kestra secrets under SLACK_WEBHOOK_URL.

Secrets

SLACK_WEBHOOK_URL: Secure webhook URL for routing security alerts to SecOps channels.

Quick start

Confirm that your container scanner is actively indexing registry images.
Add the SLACK_WEBHOOK_URL secret within your Kestra namespace settings.
Specify your target container image repository in target_image.
Trigger a manual execution via the Kestra UI to validate CVE evaluation and review the approval gate.
Enable the periodic_vulnerability_audit schedule trigger to establish automated surveillance.

Common pitfalls and troubleshooting

Approval Gate Timeout: The approval step times out after 4 hours; ensure on-call SecOps engineers are alerted via Slack.
Distro Compatibility: Copacetic supports Debian, Ubuntu, Alpine, and Red Hat/CentOS package managers; confirm base image compatibility.
Registry Push Permissions: When executing copa patch, verify that credentials permit pushing the resulting tagged image.
Firewall Egress: Ensure build runners have outbound network access to official Linux package mirrors during patching.

How to extend

Canary Promotion: Automatically deploy the patched container into a staging cluster for automated regression testing.
Jira Ticket Creation: Automatically file a tracking issue with CVE identifiers for compliance auditing.
Multi-Arch Support: Expand the hotpatch command to generate multi-architecture container manifests.

Copacetic Container Vulnerability Hotpatch Engine

How it works

Architecture diagram

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Copacetic Container Vulnerability Hotpatch Engine

How it works

Architecture diagram

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Multi-Tenant CI/CD, Promote Kestra Flows Across Environments and Tenants Automatically

Scheduled Container Registry Vulnerability Audit with Trivy and Slack Alerts

Corosync Totem Token Timeout and Quorum Loss Sentinel

Test rendered Kubernetes manifests with conftest before they merge

Lint Commit Messages Against Conventional Commits Before Every Release

HashiCorp Consul Raft Leader Stability and Quorum Sentinel

More Cloud Blueprints

Kubernetes CoreDNS Cluster DNS Resolution and Pod Health Sentinel

KMS Key Rotation Compliance Gate

Kubernetes CronJob and Batch Job Execution Failure Sentinel

Kubernetes CNI IPAM Node Pod Subnet Exhaustion Sentinel

Kong Ingress Upstream Target Unhealthy and Circuit Breaker Sentinel

Kubecost Namespace Rightsizing with Approval and Chargeback

New to Kestra?