Docker Host Image Retention and Disk Reclaim with Kestra

id: docker-host-image-retention-reclaim
namespace: company.team

description: |
  Reclaim disk on a Docker build host or self-hosted CI runner by applying an image retention
  policy: for every repository matching a filter, keep the newest N tags and every protected
  tag (prod, latest, release-*), remove the other images without --force so an image used by a
  container is never deleted, prune dangling images, and list the images again to verify what
  was removed, what was kept and how much space came back.

inputs:
  - id: docker_host
    type: STRING
    displayName: Docker host
    description: Docker daemon to clean, for example unix:///var/run/docker.sock or
      tcp://build-01:2375.
    defaults: "unix:///var/run/docker.sock"

  - id: image_filter
    type: STRING
    displayName: Image reference filter
    description: Only images matching this Docker reference filter are considered,
      for example registry.example.com/shop/*.
    defaults: "registry.example.com/shop/*"

  - id: keep_per_repository
    type: INT
    displayName: Tags kept per repository
    defaults: 3

  - id: protected_tags
    type: STRING
    displayName: Protected tag regex
    description: An image with any tag matching this regex is always kept.
    defaults: "^(latest|prod|stable|release-.*)$"

  - id: prune_dangling
    type: BOOL
    displayName: Prune dangling images
    defaults: true

  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: When true, the plan is only logged. Must be false for any image to
      be removed.
    defaults: true

tasks:
  - id: list_images
    type: io.kestra.plugin.docker.cli.ImageLs
    description: Read-only.
    host: "{{ inputs.docker_host }}"
    imageNameFilter: "{{ inputs.image_filter }}"

  - id: plan
    type: io.kestra.plugin.core.output.OutputValues
    description: >-
      Per repository, rank tags by image creation time. An image is kept if any
      of its tags is in the newest keep_per_repository of its repository or
      matches protected_tags; otherwise it is a removal candidate.
    values:
      items: >-
        {{ {'images': outputs.list_images.images ?? [], 'keep':
        inputs.keep_per_repository, 'protect': inputs.protected_tags} | jq('
           .keep as $keep | .protect as $re
           | [.images[] | . as $img | .repoTags[] | {id: $img.id, size: $img.size, created: $img.created, ref: ., repo: (split(":") | .[:-1] | join(":")), tag: (split(":") | last)}] as $tags
           | ($tags | group_by(.repo) | map(sort_by(-.created, .tag) | .[:$keep] | map(.id)) | add // []) as $newest
           | [.images[] | . as $img
              | ($img.repoTags | map(split(":") | last) | map(select(test($re)))) as $prot
              | {id: $img.id, refs: $img.repoTags, size: $img.size, created: $img.created,
                 decision: (if ($newest | index($img.id)) != null then "KEEP_NEWEST" elif ($prot | length) > 0 then "KEEP_PROTECTED" else "REMOVE" end)}]
           | sort_by(.decision, .refs[0])') | first | toJson }}

  - id: summary
    type: io.kestra.plugin.core.log.Log
    message: >-
      Image retention on {{ inputs.docker_host }} for {{ inputs.image_filter }}:
      {{ fromJson(outputs.plan.values.items) | length }} images. {{
      fromJson(outputs.plan.values.items) | jq('group_by(.decision) |
      map("\(.[0].decision) \(length)") | join(", ")') | first }}. To remove: {{
      fromJson(outputs.plan.values.items) | jq('map(select(.decision ==
      "REMOVE") | .refs | join("+")) | join(", ")') | first }} ({{
      fromJson(outputs.plan.values.items) | jq('map(select(.decision ==
      "REMOVE") | .size) | add // 0 | . / 1048576 | floor | tostring |
      sub("\\.0$"; "")') | first }} MiB at most, shared layers excluded).

  - id: apply_gate
    type: io.kestra.plugin.core.flow.If
    condition: "{{ not inputs.dry_run }}"
    then:
      - id: remove
        type: io.kestra.plugin.core.flow.Loop
        description: One docker rmi per image, all its tags at once, never forced. An
          image used by a container fails here and is reported as in use.
        values: "{{ fromJson(outputs.plan.values.items) | jq('map(select(.decision ==
          \"REMOVE\"))') | first | toJson }}"
        tasks:
          - id: rmi
            type: io.kestra.plugin.docker.cli.Rm
            allowFailure: true
            host: "{{ inputs.docker_host }}"
            imageIds: "{{ fromJson(item.value).refs }}"
            force: false

      - id: prune
        type: io.kestra.plugin.docker.cli.Prune
        runIf: "{{ inputs.prune_dangling }}"
        host: "{{ inputs.docker_host }}"
        pruneType: IMAGES
        dangling: true

      - id: relist
        type: io.kestra.plugin.docker.cli.ImageLs
        host: "{{ inputs.docker_host }}"
        imageNameFilter: "{{ inputs.image_filter }}"

      - id: result
        type: io.kestra.plugin.core.output.OutputValues
        values:
          r: >-
            {{ {'plan': fromJson(outputs.plan.values.items), 'after':
            outputs.relist.images ?? []} | jq('
               (.after | map(.id)) as $a
               | [.plan[] | .id as $i | . + {present: (($a | index($i)) != null)}] as $p
               | {removed: [$p[] | select(.decision == "REMOVE" and (.present | not))],
                  in_use: [$p[] | select(.decision == "REMOVE" and .present)],
                  kept_missing: [$p[] | select(.decision != "REMOVE" and (.present | not))]}') | first | toJson }}

      - id: verify_result
        type: io.kestra.plugin.core.flow.If
        description: A kept image must never be gone. A candidate still present is only
          reported, it is used by a container.
        condition: "{{ fromJson(outputs.result.values.r).kept_missing | length > 0 }}"
        then:
          - id: kept_image_removed
            type: io.kestra.plugin.core.execution.Fail
            errorMessage: "Images the policy keeps are missing after the run: {{
              fromJson(outputs.result.values.r).kept_missing | jq('map(.refs |
              join(\"+\")) | join(\", \")') | first }}."
        else:
          - id: reclaimed
            type: io.kestra.plugin.core.log.Log
            message: >-
              Removed {{ fromJson(outputs.result.values.r).removed | length }}
              images ({{ fromJson(outputs.result.values.r).removed | jq('if
              length == 0 then "none" else map(.refs | join("+")) | join(", ")
              end') | first }}), up to {{
              fromJson(outputs.result.values.r).removed | jq('map(.size) | add
              // 0 | . / 1048576 | floor | tostring | sub("\\.0$"; "")') | first
              }} MiB. Still in use by a container, kept: {{
              fromJson(outputs.result.values.r).in_use | jq('if length == 0 then
              "none" else map(.refs | join("+")) | join(", ") end') | first }}.
              Dangling images pruned: {{ inputs.prune_dangling }}.

          - id: in_use_notice
            type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
            runIf: "{{ fromJson(outputs.result.values.r).in_use | length > 0 }}"
            url: "{{ secret('SLACK_WEBHOOK_URL') }}"
            payload: |
              {{ {'text': 'Image retention on ' ~ inputs.docker_host ~ ': removed ' ~ (fromJson(outputs.result.values.r).removed | length) ~ ' images. Kept because a container still uses them: ' ~ (fromJson(outputs.result.values.r).in_use | jq('map(.refs | join("+")) | join(", ")') | first) ~ '. Check for forgotten containers.'} | toJson }}
    else:
      - id: dry_run_only
        type: io.kestra.plugin.core.log.Log
        message: "Dry run: nothing removed. Run with dry_run false to apply this plan."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {'text': 'Docker image retention failed on ' ~ inputs.docker_host ~ ' (execution ' ~ execution.id ~ '). Disk was not reclaimed, check the logs.'} | toJson }}

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    description: Shipped disabled. Runs in dry-run mode until you change it.
    cron: "30 2 * * *"
    disabled: true
    inputs:
      dry_run: true

Self-hosted CI runners and build hosts fill their disks with images: one per commit, per branch, per service, kept forever. docker image prune -a frees the space but also deletes the base images the next build needs, and the tag someone deployed by hand. So the disk alarm fires every few weeks, and someone deletes images by hand on a production runner.

This blueprint applies a retention policy instead. For every repository matching a filter, it keeps the newest N tags and any image with a protected tag, and removes the other images one by one without --force. An image a container still uses is therefore never deleted. It prunes dangling images, then lists the images again to verify that nothing the policy keeps went missing.

This blueprint was created by zkasuran.

How it works

list_images (io.kestra.plugin.docker.cli.ImageLs, read-only) lists the images matching image_filter.
plan groups their tags by repository and ranks them by creation time:
- KEEP_NEWEST: one of the newest keep_per_repository of its repository;
- KEEP_PROTECTED: one of its tags matches protected_tags;
- otherwise REMOVE.
summary logs the plan and the space it frees at most.
apply_gate (io.kestra.plugin.core.flow.If) stops there when dry_run is true. Otherwise:
- remove (io.kestra.plugin.core.flow.Loop over io.kestra.plugin.docker.cli.Rm, force: false) removes each candidate with all its tags. A failure, usually an image in use, does not stop the others.
- prune (io.kestra.plugin.docker.cli.Prune, IMAGES, dangling: true) removes untagged layers.
- relist and result compare the images before and after. verify_result fails the run if any kept image is missing.
- reclaimed logs what was removed. in_use_notice posts to Slack when a candidate was kept because a container uses it, often a forgotten container.
errors alerts Slack when the flow fails, for example on an unreachable daemon.

What you get

A retention policy per repository, instead of an all-or-nothing prune.
Protected tags and in-use images that are never removed, by design rather than by luck.
A before and after comparison on every run, and a notice for forgotten containers holding old images.

Who it's for

Teams running self-hosted CI runners (GitHub Actions, GitLab, Buildkite, Jenkins agents) with a local Docker daemon.
Platform teams operating shared build hosts or Docker-based edge nodes.
Teams evaluating the Docker plugin who want an example of ImageLs, Rm and Prune driven by a plan.

Why orchestrate this with Kestra

A cron job running docker image prune cannot tell a release tag from a commit build, or report what it deleted. Kestra lists the images, computes the plan in one jq expression, gates the change behind dry_run, removes images one task at a time so one failure is isolated, verifies the result, and keeps an execution history of what was removed from which host.

Tested end to end

On Kestra 2.0.5 OSS against an isolated Docker 27 daemon (Docker-in-Docker), with a local HTTP sink standing in for Slack. The daemon held:

shop/api tags 1.0.1 to 1.0.7, with 1.0.2 also tagged prod, and 1.0.3 used by a running container;
shop/worker tags 2.1 to 2.3;
one dangling image, plus the alpine and busybox base images.

Run	Inputs	Result
1	defaults (dry run)	10 images: 6 newest kept, `1.0.2` kept as protected, 3 to remove (`1.0.1`, `1.0.3`, `1.0.4`). Nothing removed
2	`dry_run: false`	`1.0.1` and `1.0.4` removed. `1.0.3` kept because a container uses it, and the Slack notice is sent. Dangling image pruned. Verified: every kept image is still there
3	`dry_run: false` again	Nothing new to remove, `1.0.3` still reported in use
4	any run	`alpine` and `busybox` untouched: they do not match the filter
5	unreachable daemon	`errors` alert sent

Prerequisites

Access to the Docker daemon to clean, through its socket mounted into the Kestra worker or a TCP endpoint.
Pick an image_filter that matches only the images your builds produce.
Local testing: docker run -d --privileged --name dind -e DOCKER_TLS_CERTDIR= docker:27-dind, then build a few tagged images in it, and point docker_host at tcp://dind:2375.

Secrets

SLACK_WEBHOOK_URL: used by in_use_notice and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
`docker_host`	`unix:///var/run/docker.sock`	Daemon to clean
`image_filter`	`registry.example.com/shop/*`	Images considered
`keep_per_repository`	`3`	Newest tags kept per repository
`protected_tags`	`^(latest\|prod\|stable\|release-.*)$`	Tags always kept
`prune_dangling`	`true`	Also prune untagged images
`dry_run`	`true`	Must be `false` for anything to be removed

Quick start

Add the SLACK_WEBHOOK_URL secret.
Import the flow and set docker_host and image_filter for one build host.
Run it with the defaults and read the plan in the logs.
Run it with dry_run: false to apply the plan.
Enable the nightly trigger, and change its dry_run input to false once the plans look right.

How to extend

Loop over a list of build hosts to clean every runner in one execution.
Trigger the flow from your disk usage alert instead of a schedule.
Add Prune with BUILD to clear the build cache, and CONTAINERS with until to remove old stopped containers.
Keep the tags currently deployed by reading them from your deployment tool and adding them to protected_tags.

Docker Host Image Retention and Disk Reclaim

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Docker Host Image Retention and Disk Reclaim

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Block a release when k6 shows it is slower than the last good one

Kubernetes Pod Lifecycle Management: Create, Inspect, and Cleanup

Kafka Consumer Group Offset Checkpoint and Rewind

Self-heal Kafka Connect connectors with a restart budget and escalation

Kafka Client Quota Guard

Kafka ACL Overly Broad Grant Audit

More Core Blueprints

Purge KV Store

Layered Error Handling with Retries and KV Context

Kubernetes Node Resource Pressure Sentinel

LDAP Employee Offboarding and Account Deprovisioning

Block a release when k6 shows it is slower than the last good one

LDAP Privileged Group Membership Gate

New to Kestra?