Schedule icon
SkopeoCLI icon
Script icon
Switch icon
Log icon
SlackIncomingWebhook icon

Audit Container Image Multi-Arch Manifest Compliance with Skopeo

Audit container registry multi-arch manifest compliance with Skopeo and Kestra. Verify amd64 and arm64 coverage and dispatch automated Slack alerts.

Categories
DataInfrastructure

In heterogeneous Kubernetes environments featuring mixed worker node architectures (such as x86_64 Intel/AMD and ARM64 AWS Graviton or Apple Silicon nodes), container workloads must provide multi-architecture manifest lists. If a release pipeline or third-party vendor publishes an image that is single-architecture (for example, only compiled for amd64), scheduling that pod onto an arm64 worker node results in immediate CrashLoopBackOff or exec format errors.

This blueprint provides an automated Day 2 Operations governance gate that uses Skopeo to inspect remote container registry manifests without downloading heavy image layers. It parses OCI and Docker manifest indexes, verifies the presence of required CPU architectures, and alerts DevOps teams over Slack before faulty images cause cluster rollout failures.

How it works

  1. schedule (io.kestra.plugin.core.trigger.Schedule) initiates the audit cycle every 6 hours (disabled by default for safe onboarding).
  2. inspect_manifest (io.kestra.plugin.skopeo.cli.SkopeoCLI) queries the container registry directly using skopeo inspect --raw without pulling layers, saving the manifest index as manifest.json.
  3. classify_manifest_compliance (io.kestra.plugin.scripts.python.Script) parses the manifest JSON, evaluates platform entries against required_architectures (e.g. amd64,arm64) and required_os (linux), and outputs a typed status (COMPLIANT vs NON_COMPLIANT).
  4. evaluate_compliance (io.kestra.plugin.core.flow.Switch) routes execution based on the compliance verdict.
  5. When compliant (COMPLIANT), log_compliant_status logs verified architectures.
  6. When non-compliant (NON_COMPLIANT), alert_non_compliant_manifest dispatches a formatted alert to Slack using io.kestra.plugin.slack.notifications.SlackIncomingWebhook, and log_remediation_guidance logs actionable build remediation advice.
  7. log_audit_completion writes a final operational audit summary.
  8. The flow-level errors block intercepts unexpected network or registry authentication failures and alerts Slack.

What you get

  • Zero-download registry inspection querying image manifests in milliseconds.
  • Automated detection of missing CPU architectures (amd64, arm64, s390x, ppc64le).
  • Protection against mixed-node Kubernetes scheduling failures and exec format errors.
  • Automated Slack incident notifications with missing platform entries and execution context.
  • Flow-level error alerting safeguarding pipeline reliability.

Who it's for

  • Platform Engineers and DevSecOps Teams maintaining multi-architecture Kubernetes clusters.
  • Release Engineers responsible for container image promotion gates across staging and production.
  • Site Reliability Engineers safeguarding container supply-chain integrity.

Why orchestrate this with Kestra

Inspecting remote registry manifests with standalone shell scripts or CI job steps requires maintaining Skopeo installations, handling edge cases across Docker vs OCI index specifications, and managing webhook credentials across multiple repositories. Kestra provides a unified orchestration platform with native Skopeo container sandboxing, structured JSON evaluation, conditional branching, execution lineage, and centralized Slack alerting.

Prerequisites

  • Docker available to the Kestra Worker for running SkopeoCLI (quay.io/skopeo/stable).
  • Network access from the Kestra Worker to the target container registry.
  • A Slack Incoming Webhook URL configured in Kestra secrets.

Local Container Setup

To run a local container registry for development and testing while avoiding host port 8080 conflicts:

docker run -d --name local-registry -p 5001:5000 registry:2

If Kestra is running inside a Docker network, connect the registry container to Kestra network:

docker network ls
docker network connect YOUR_KESTRA_NETWORK local-registry

Secrets

  • SLACK_WEBHOOK: Incoming Webhook URL for Slack alerting.
  • REGISTRY_USER: Optional username for private container registries.
  • REGISTRY_PASSWORD: Optional password or token for private container registries.

Note for Kestra Open-Source (OSS): Secrets are provided as base64-encoded environment variables formatted as SECRET_<NAME> on the Kestra server or worker environment.

Inputs

Name Type Default Allowed Values Description
image_reference STRING "registry.hub.docker.com/library/python:3.11-slim" Any valid image reference Target image to inspect in registry
required_architectures STRING "amd64,arm64" Comma-separated archs Architectures required in manifest list
required_os STRING "linux" Operating system name Expected OS for platform entries
remediation_mode SELECT "ALERT_ONLY" "ALERT_ONLY", "QUARANTINE_LOG" Operational remediation action mode
dry_run SELECT "true" "true", "false" Dry run simulation flag

Outputs

  • {{ outputs.inspect_manifest.outputFiles['manifest.json'] }}: Raw manifest index JSON file URI.
  • {{ outputs.classify_manifest_compliance.vars.status }}: Compliance status (COMPLIANT or NON_COMPLIANT).
  • {{ outputs.classify_manifest_compliance.vars.found_architectures }}: Array of detected architectures matching required OS.
  • {{ outputs.classify_manifest_compliance.vars.missing_architectures }}: Array of missing required architectures.

Quick start

  1. Configure SLACK_WEBHOOK in your Kestra namespace secrets.
  2. Import this blueprint into your Kestra instance.
  3. Run a manual execution with default inputs to inspect the official python:3.11-slim multi-arch image.
  4. Verify execution logs and confirm that both amd64 and arm64 platforms are verified.
  5. Enable the schedule trigger to audit your critical container images periodically.

How to extend

  • Pass authentication flags (--src-creds {{ secret('REGISTRY_USER') }}:{{ secret('REGISTRY_PASSWORD') }}) to skopeo inspect for private registries.
  • Chain a downstream Kubernetes rollout or Helm deploy subflow via io.kestra.plugin.core.flow.Subflow that only executes when the gate passes.
  • Loop through an inventory of production container images using io.kestra.plugin.core.flow.Loop.

Pitfalls

  • Single-Arch vs Multi-Arch Manifests: Standard single-architecture images return a single manifest rather than a manifest list, which this flow flags as non-compliant if multi-arch support is required.
  • Private Registry Authentication: Private registries require credentials passed as --creds or --src-creds in the commands array.
  • Port Conflicts: Ensure local test registry port 5001 does not collide with local services.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.