Schedule icon
SkopeoCLI icon
OutputValues icon
If icon
SlackIncomingWebhook icon
Fail icon
Log icon

Skopeo Container Image Digest Drift and Tag Immutability Sentinel

Audit remote container image manifests with Skopeo, detect silent tag digest drift or overwrites, and alert Slack to block untracked container deployments.

Categories
Infrastructure

Continuously verify remote OCI container image immutability without pulling layers, catch silent tag overwrites or digest drift, and enforce deployment gates before untracked binaries reach production clusters.

In modern container orchestration, deploying mutable tags (such as :latest, :staging, or release versions like :v1.2.0) introduces significant supply chain security risks. If an upstream registry tag is overwritten, whether through an untracked rebuild, CI misconfiguration, or unauthorized push, nodes can pull unverified binaries without triggering standard CI/CD checks. This blueprint orchestrates skopeo inspect through Kestra's official io.kestra.plugin.skopeo.cli.SkopeoCLI task to inspect remote manifests directly over the network, comparing the resolved SHA256 digest against approved baselines.

How it works

  1. scheduled_immutability_audit (io.kestra.plugin.core.trigger.Schedule) initiates regular drift detection sweeps every 4 hours.
  2. inspect_remote_image (io.kestra.plugin.skopeo.cli.SkopeoCLI) queries the target container registry (docker://<image_reference>) and streams raw manifest JSON to inspect_metadata.json without pulling heavy container layers.
  3. parse_image_digest (io.kestra.plugin.core.output.OutputValues) extracts the resolved Digest, Architecture, Os, and Created timestamps using jq filtering.
  4. evaluate_digest_drift (io.kestra.plugin.core.output.OutputValues) compares the observed digest against expected_digest, classifying the outcome into IMMUTABLE, DIGEST_DRIFT, or UNREACHABLE.
  5. route_integrity_findings (io.kestra.plugin.core.flow.If) evaluates the integrity status:
    • If drift or registry outage is detected, alert_digest_drift (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) dispatches a high-priority incident payload to Slack with full OCI manifest details.
    • If enforce_immutability is enabled, fail_on_drift (io.kestra.plugin.core.execution.Fail) halts execution, blocking downstream deployment pipelines.
    • If the digest matches, log_immutability_verified (io.kestra.plugin.core.log.Log) writes an immutable verification receipt to the execution audit trail.

What you get

  • Zero-bandwidth container inspection: verifies remote image metadata and digests without pulling multi-gigabyte layers.
  • Automated detection of silent tag mutability, unauthorized registry overwrites, or corrupted manifest lists.
  • Structured Slack notifications detailing exact image references, observed vs expected digests, and creation timestamps.
  • An automated deployment gate that halts execution before mutated container images reach staging or production Kubernetes clusters.

Who it's for

  • Platform and Site Reliability Engineers (SREs) safeguarding production Kubernetes and Nomad clusters.
  • DevSecOps teams enforcing software supply chain security and SLSA provenance gates.
  • Release engineers validating container image promotion across environments.

Why orchestrate this with Kestra

  • Agentless Registry Inspection: The Skopeo CLI wrapper executes directly within containerized tasks, querying public or private registries without needing Docker daemon privileges.
  • Pebble & JQ Expression Engine: Native expression evaluation parses complex OCI manifest trees and hashes directly in memory.
  • Governed Deployment Gates: Easily embedded as a pre-flight subflow in deployment pipelines to gate production rollouts behind cryptographic digest validation.
  • Declarative Audit History: Every execution captures complete JSON manifests and audit receipts in Kestra internal storage.

Prerequisites

  • A container runtime (Docker) available on the Kestra worker to run quay.io/skopeo/stable:latest.
  • Network reachability from the Kestra worker to the target container registry (e.g., Docker Hub, Quay, GHCR, or private ECR/GAR).
  • A Slack incoming webhook URL for incident notifications.

Secrets

  • SLACK_WEBHOOK_URL: The Slack incoming webhook URL for platform alerts and supply-chain drift incidents.

Quick start

  1. Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace.
  2. Import this blueprint into your Kestra instance under namespace company.platform.
  3. Run a manual test execution with default inputs (AUTO_DISCOVER mode) to inspect the current remote digest.
  4. Copy the verified SHA256 digest into expected_digest and activate the schedule to enforce continuous immutability.

How to extend

  • Integrate Private Registry Authentication: Add registry credentials via --creds {{ secret('REGISTRY_USER') }}:{{ secret('REGISTRY_TOKEN') }} inside the skopeo inspect command flags.
  • Multi-Region Registry Mirror Check: Query secondary registry mirrors in parallel and assert digest parity across global clusters.
  • Automated Quarantine Webhook: Trigger a downstream Kubernetes admission controller or image security scanner when drift is detected.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.