Docker BuildKit Build Cache Ceiling and Reclaim Guard with Kestra

id: docker-build-cache-reclaim-guard
namespace: company.team

description: >-
  Guard the BuildKit build cache on one Docker build host with a high watermark
  that triggers and a most-recently-used floor that survives. The flow reads the
  daemon's own build cache records, works out how much of the cache is shared
  with image layers and cannot be released, reclaims down to the floor behind an
  approval gate, then reads the records again and fails the run if the cache is
  still over the ceiling or if a record inside the floor was evicted. A BuildKit
  prune is daemon global, so point this at a single build host.

labels:
  system: docker
  surface: buildkit

inputs:
  - id: docker_host
    type: STRING
    displayName: Docker host
    description: >-
      The daemon whose build cache is guarded, reached over TCP, for example
      tcp://build-01:2375. A BuildKit prune cannot be scoped by label, so it
      applies to every build on this daemon. Never point this at a shared or
      production daemon.
    defaults: "tcp://build-01:2375"

  - id: api_version
    type: STRING
    displayName: Docker Engine API version
    description: >-
      Pinned API version used for the two daemon calls. Docker 28 renamed the
      prune parameter, so bump this only together with the parameter name on the
      reclaim task.
    defaults: "v1.47"

  - id: max_cache_mb
    type: INT
    displayName: Build cache ceiling in MiB
    description: Total build cache above this size triggers the flow. Nothing
      happens below it.
    defaults: 20480

  - id: mru_keep_mb
    type: INT
    displayName: Most-recently-used floor in MiB
    description: >-
      The reclaim stops here. BuildKit evicts least-recently-used records first,
      so this many MiB of the cache the builds touched last stays in place. Must
      be below the ceiling.
    defaults: 8192

  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: True reports the reclaim without calling the daemon's prune endpoint.
    defaults: true

  - id: auto_remediate
    type: BOOL
    displayName: Auto remediate
    description: False keeps the flow read-only. A reclaim needs this true, dry_run
      false plus an approval.
    defaults: false

tasks:
  - id: check_watermarks
    type: io.kestra.plugin.core.flow.If
    description: A floor at or above the ceiling can never satisfy the ceiling, so
      stop before touching the daemon.
    condition: "{{ inputs.mru_keep_mb >= inputs.max_cache_mb }}"
    then:
      - id: watermarks_inverted
        type: io.kestra.plugin.core.execution.Fail
        errorMessage: >-
          mru_keep_mb is {{ inputs.mru_keep_mb }} MiB and max_cache_mb is {{
          inputs.max_cache_mb }} MiB. The floor must be below the ceiling.

  - id: read_cache
    type: io.kestra.plugin.core.http.Request
    description: Read-only. Every build cache record on the daemon with its size,
      last used time plus whether an image layer shares it.
    uri: "{{ inputs.docker_host | replace({'tcp://': 'http://'}) }}/{{
      inputs.api_version }}/system/df?type=build-cache"
    method: GET

  - id: list_images
    type: io.kestra.plugin.docker.cli.ImageLs
    description: Read-only. Images on the same daemon, used to explain the share of
      the cache a prune cannot release.
    host: "{{ inputs.docker_host }}"
    showAll: true

  - id: plan
    type: io.kestra.plugin.core.output.OutputValues
    description: >-
      Size the cache, split it into the part a prune can release and the part
      image layers pin, then rank the releasable records by last used time and
      mark the ones inside the floor as protected.
    values:
      p: >-
        {{ {'df': outputs.read_cache.body, 'ceiling': inputs.max_cache_mb *
        1048576, 'floor': inputs.mru_keep_mb * 1048576} | toJson | jq('
           (.df | fromjson) as $d | .ceiling as $ceiling | .floor as $floor
           | ($d.BuildCache // []) as $r
           | ([$r[].Size] | add // 0) as $total
           | ([$r[] | select(.Shared == true) | .Size] | add // 0) as $pinned
           | ([$r[] | select(.Shared != true)] | sort_by(.LastUsedAt // .CreatedAt // "") | reverse) as $mru
           | ([$mru[].Size] | add // 0) as $releasable
           | [foreach $mru[] as $x (0; . + $x.Size; {id: $x.ID, used: ($x.LastUsedAt // $x.CreatedAt // ""), size: $x.Size, cum: .})] as $cum
           | [$cum[] | select(.cum <= $floor)] as $prot
           | (if $releasable < $floor then $releasable else $floor end) as $kept
           | ($pinned + $kept) as $predicted
           | {records: ($r | length), total_bytes: $total, pinned_bytes: $pinned,
              pinned_records: ([$r[] | select(.Shared == true)] | length),
              releasable_bytes: $releasable, ceiling_bytes: $ceiling, floor_bytes: $floor,
              over_bytes: (if $total > $ceiling then $total - $ceiling else 0 end),
              release_bytes: ($releasable - $kept), predicted_bytes: $predicted,
              mru_id: ($mru[0].ID // ""), mru_used: ($mru[0].LastUsedAt // ""),
              protected_ids: [$prot[] | .id], protected_bytes: ([$prot[] | .cum] | max // 0),
              ceiling_reachable: ($predicted <= $ceiling)}') | first | toJson }}

  - id: report
    type: io.kestra.plugin.core.log.Log
    message: >-
      Build cache on {{ inputs.docker_host }}: {{
      fromJson(outputs.plan.values.p).records }} records, {{
      fromJson(outputs.plan.values.p).total_bytes }} bytes against a ceiling of
      {{ fromJson(outputs.plan.values.p).ceiling_bytes }} bytes. {{
      fromJson(outputs.plan.values.p).pinned_bytes }} bytes in {{
      fromJson(outputs.plan.values.p).pinned_records }} records are shared with
      image layers ({{ outputs.list_images.images | length }} images on this
      daemon), so a cache prune cannot release them. Releasable now: {{
      fromJson(outputs.plan.values.p).releasable_bytes }} bytes. Over the
      ceiling by {{ fromJson(outputs.plan.values.p).over_bytes }} bytes. Floor
      {{ fromJson(outputs.plan.values.p).floor_bytes }} bytes protects {{
      fromJson(outputs.plan.values.p).protected_ids | length }} records ({{
      fromJson(outputs.plan.values.p).protected_bytes }} bytes), most recent {{
      fromJson(outputs.plan.values.p).mru_id }} last used {{
      fromJson(outputs.plan.values.p).mru_used }}.

  - id: over_ceiling
    type: io.kestra.plugin.core.flow.If
    condition: "{{ fromJson(outputs.plan.values.p).over_bytes > 0 }}"
    then:
      - id: pinned_blocks_ceiling
        type: io.kestra.plugin.core.log.Log
        runIf: "{{ not fromJson(outputs.plan.values.p).ceiling_reachable }}"
        level: WARN
        message: >-
          Image layers pin {{ fromJson(outputs.plan.values.p).pinned_bytes }}
          bytes of cache, which a build cache prune cannot release. Even a full
          reclaim down to the floor leaves {{
          fromJson(outputs.plan.values.p).predicted_bytes }} bytes against a
          ceiling of {{ fromJson(outputs.plan.values.p).ceiling_bytes }} bytes.
          Remove the images that hold those layers first.

      - id: planned_reclaim
        type: io.kestra.plugin.core.log.Log
        message: >-
          Reclaim plan: drop least-recently-used records until the releasable
          pool fits {{ fromJson(outputs.plan.values.p).floor_bytes }} bytes,
          which releases {{ fromJson(outputs.plan.values.p).release_bytes }}
          bytes and leaves about {{
          fromJson(outputs.plan.values.p).predicted_bytes }} bytes, keeping {{
          fromJson(outputs.plan.values.p).protected_ids | length }} protected
          records in place.

      - id: read_only_notice
        type: io.kestra.plugin.core.log.Log
        runIf: "{{ inputs.dry_run or not inputs.auto_remediate }}"
        message: >-
          Reported only. dry_run is {{ inputs.dry_run }} plus auto_remediate is
          {{ inputs.auto_remediate }}, so the daemon prune endpoint was not
          called. No cache record was removed.

      - id: reclaim_gate
        type: io.kestra.plugin.core.flow.If
        condition: "{{ inputs.auto_remediate and not inputs.dry_run }}"
        then:
          - id: approve_reclaim
            type: io.kestra.plugin.core.flow.Pause
            description: >-
              A BuildKit prune cannot be undone. Resume with approved true to
              reclaim, approved false to walk away. An unanswered resume reads
              as false.
            onResume:
              - id: approved
                type: BOOL
                displayName: Reclaim the build cache on this daemon
                defaults: false

          - id: apply_reclaim
            type: io.kestra.plugin.core.flow.If
            condition: "{{ outputs.approve_reclaim.onResume.approved ?? false }}"
            then:
              - id: reclaim_build_cache
                type: io.kestra.plugin.core.http.Request
                description: >-
                  The daemon's own build cache prune, floored. keep-storage is
                  the only interface that keeps a most-recently-used working
                  set. It applies to the releasable pool only, so BuildKit drops
                  unused records least-recently-used first until that pool fits
                  the floor. Records an image layer shares are left alone.
                uri: "{{ inputs.docker_host | replace({'tcp://': 'http://'}) }}/{{
                  inputs.api_version }}/build/prune?keep-storage={{
                  fromJson(outputs.plan.values.p).floor_bytes }}"
                method: POST

              - id: reclaim_result
                type: io.kestra.plugin.core.log.Log
                message: >-
                  Daemon reclaimed {{ outputs.reclaim_build_cache.body |
                  jq('.SpaceReclaimed') | first }} bytes across {{
                  outputs.reclaim_build_cache.body | jq('.CachesDeleted |
                  length') | first }} cache records.

              - id: reread_cache
                type: io.kestra.plugin.core.http.Request
                description: Read the records again. The prune reporting success says nothing
                  about the size that is left.
                uri: "{{ inputs.docker_host | replace({'tcp://': 'http://'}) }}/{{
                  inputs.api_version }}/system/df?type=build-cache"
                method: GET

              - id: verify
                type: io.kestra.plugin.core.output.OutputValues
                values:
                  v: >-
                    {{ {'df': outputs.reread_cache.body, 'ceiling':
                    inputs.max_cache_mb * 1048576,
                        'before': fromJson(outputs.plan.values.p).total_bytes,
                        'protected': fromJson(outputs.plan.values.p).protected_ids} | toJson | jq('
                       (.df | fromjson) as $d | .ceiling as $ceiling | .before as $before
                       | ($d.BuildCache // []) as $r | [$r[].ID] as $ids
                       | ([$r[].Size] | add // 0) as $after
                       | {after_bytes: $after, records: ($r | length), freed_bytes: ($before - $after),
                          under_ceiling: ($after <= $ceiling),
                          kept: [.protected[] | select(. as $p | ($ids | index($p)) != null)],
                          lost: [.protected[] | select(. as $p | ($ids | index($p)) == null)]}') | first | toJson }}

              - id: verify_result
                type: io.kestra.plugin.core.log.Log
                message: >-
                  After the reclaim: {{
                  fromJson(outputs.verify.values.v).records }} records, {{
                  fromJson(outputs.verify.values.v).after_bytes }} bytes, freed
                  {{ fromJson(outputs.verify.values.v).freed_bytes }} bytes.
                  Under ceiling: {{
                  fromJson(outputs.verify.values.v).under_ceiling }}. Protected
                  records still present: {{
                  fromJson(outputs.verify.values.v).kept | length }} of {{
                  fromJson(outputs.plan.values.p).protected_ids | length }}.

              - id: verify_gate
                type: io.kestra.plugin.core.flow.If
                description: The run fails if the cache is still over the ceiling or if the
                  floor did not hold.
                condition: "{{ (fromJson(outputs.verify.values.v).under_ceiling == false) or
                  (fromJson(outputs.verify.values.v).lost | length > 0) }}"
                then:
                  - id: reclaim_did_not_land
                    type: io.kestra.plugin.core.execution.Fail
                    errorMessage: >-
                      Build cache on {{ inputs.docker_host }} is {{
                      fromJson(outputs.verify.values.v).after_bytes }} bytes
                      after the reclaim against a ceiling of {{
                      fromJson(outputs.plan.values.p).ceiling_bytes }} bytes,
                      with {{ fromJson(outputs.verify.values.v).lost | length }}
                      protected records lost. {{
                      fromJson(outputs.plan.values.p).pinned_bytes }} bytes are
                      pinned by image layers.
                else:
                  - id: reclaim_landed
                    type: io.kestra.plugin.core.log.Log
                    message: >-
                      Build cache is back under the ceiling at {{
                      fromJson(outputs.verify.values.v).after_bytes }} bytes
                      with the floor intact.
            else:
              - id: reclaim_declined
                type: io.kestra.plugin.core.log.Log
                message: Reclaim declined at the approval gate. The build cache was not touched.
    else:
      - id: under_ceiling
        type: io.kestra.plugin.core.log.Log
        message: >-
          Build cache is {{ fromJson(outputs.plan.values.p).total_bytes }}
          bytes, under the ceiling of {{
          fromJson(outputs.plan.values.p).ceiling_bytes }} bytes. Nothing to
          reclaim.

errors:
  - id: alert_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {'text': 'Build cache guard failed on ' ~ inputs.docker_host ~ ' (execution ' ~ execution.id ~ '). The cache may still be over the ceiling, check the logs.'} | toJson }}

triggers:
  - id: nightly
    type: io.kestra.plugin.core.trigger.Schedule
    description: Shipped disabled. Reports in read-only mode until you change the inputs.
    cron: "20 3 * * *"
    disabled: true
    inputs:
      dry_run: true
      auto_remediate: false

BuildKit cache is usually the largest thing on a CI host. It grows without a bound. docker builder prune is the only lever most teams reach for. That lever is blind: with no flags it keeps everything a build still references, with -a it throws away the cache the next build needs. So the disk alarm fires, someone flushes the whole cache by hand, every pipeline then runs cold for an hour.

This blueprint puts two watermarks around that cache. A ceiling decides when anything happens at all. A most-recently-used floor decides what survives. Between the two it reclaims least-recently-used records only, behind an approval gate, then reads the daemon again and fails the run if the cache is still over the ceiling or if a record inside the floor was evicted.

This blueprint was created by zkasuran.

Safety

A BuildKit prune cannot be scoped by label. There is no per-project, per-branch or per-team build cache: the daemon holds one cache and a prune applies to all of it. So this flow is only safe pointed at a build host you own end to end. Point it at a daemon shared with other teams, with production containers, with another CI tenant: an approved run takes their cache too. There is no undo. docker_host is a single daemon on purpose.

How it works

check_watermarks refuses a floor at or above the ceiling before any daemon call, because that configuration can never satisfy the ceiling.
read_cache reads /system/df?type=build-cache, which returns every cache record with its size, created time, last used time, usage count plus whether an image layer shares it.
list_images lists the images on the same daemon through the Docker plugin.
plan sizes the cache, splits it into the part a prune can release and the part image layers pin, ranks the releasable records by last used time, then marks the records that fit inside the floor as protected.
report prints the numbers, including how much of the cache a prune cannot touch.
Under the ceiling, the flow logs that and stops.
Over the ceiling in read-only mode, it prints the reclaim plan plus the bytes it would release. It calls nothing.
With auto_remediate true plus dry_run false, approve_reclaim pauses. On an approval, reclaim_build_cache calls the daemon's build cache prune with keep-storage set to the floor, so BuildKit drops unused records least-recently-used first until the releasable pool fits that floor.
reread_cache reads the records again. verify compares the new total against the ceiling and checks every protected record is still there. verify_gate fails the run when either is wrong.

What you get

A cache guard that triggers on size, not on a cron that flushes everything every night.
A floor expressed in MiB of most-recently-used releasable cache, so the layers the last builds touched stay warm. BuildKit orders eviction by last used, not by created, so a record built days ago that today's build still hits survives.
The share of the cache that image layers pin, reported before the prune. This is the number that makes a naive prune look broken: docker system df counts a shared record as zero bytes while docker buildx du counts it in full. A cache prune can release none of it.
A read-back that fails the run, so a prune that reported success while the disk stayed full does not pass as green.
An approval gate on the one irreversible step.

Who it's for

Platform teams running self-hosted CI runners where BuildKit cache is the top disk consumer.
Anyone who has flushed a build cache by hand at 2am then watched every pipeline run cold.
Teams that want a cache budget per build host instead of a nightly prune -a.

Why orchestrate this with Kestra

The reclaim is a sequence, not a command: measure, classify, decide, pause for a human, write, measure again, fail if the write did not land. A cron calling docker builder prune has nowhere to put the floor, nowhere to put the approval plus no way to tell you it ran and achieved nothing. Here the inputs carry the policy, the If tasks carry the decision, the Pause carries the human, the re-read carries the proof. Kestra keeps the measurements as outputs, so the before and after numbers of every run stay attached to that run.

Tested end to end

Run against a real Docker 27.5.1 daemon on Kestra 2.0.5, with BuildKit cache generated by throwaway builds of 5 MB random layers. Every number below was read back with docker buildx du plus docker system df inside that daemon.

Under the ceiling: 4 records, 10,000,040 bytes against a 48 MiB ceiling. Zero action.
Over the ceiling, read-only: 10,000,040 bytes against an 8 MiB ceiling, reported 1,611,432 bytes over plus 5,805,736 releasable, called nothing. The daemon read 10 MB before plus after.
Approved reclaim: 9 records, 35,000,040 bytes against a 20 MiB ceiling with a 10 MiB floor. The daemon released 25,000,000 bytes across 5 records, the read-back found 4 records at 10,000,040 bytes, under the ceiling.
Declined approval: zero bytes released, 4 records at 10,000,040 bytes before plus after.
The floor held: the 4 protected ids survived the approved reclaim, confirmed by id in docker buildx du --verbose. Three of them were the oldest records in the set by creation time, kept because the last build touched them, while a record created 12 minutes later was evicted.
Re-verify failure: with 4 images pinning 20,000,000 bytes of cache, a 12 MiB ceiling plus a 5 MiB floor released only the 5,000,000 bytes that were free. The cache stayed at 25,000,040 bytes, the run failed, the Slack error block fired.
Inverted watermarks (floor at or above the ceiling) failed before any daemon call.

Prerequisites

Secrets

SLACK_WEBHOOK_URL: incoming webhook for the failure alert.

Inputs

docker_host: the build host daemon, over TCP, for example tcp://build-01:2375. The flow reads /system/df plus calls /build/prune, so it needs the Engine API reachable. A Unix socket works for the Docker plugin task but not for the two API calls.
api_version: pinned Engine API version, v1.47 by default.
max_cache_mb: the ceiling. Nothing happens below it.
mru_keep_mb: the floor the reclaim stops at. Must be below the ceiling.
dry_run: true by default.
auto_remediate: false by default.

Quick start

Add the SLACK_WEBHOOK_URL secret.
Set docker_host to one build host you own. Read the Safety note first.
Run it as shipped. It reports the cache size, the pinned share plus the reclaim it would make. Nothing changes.
Set the ceiling plus the floor to your disk budget. A floor around a third of the ceiling keeps recent layers warm without letting the cache sit at the limit.
To reclaim, run with auto_remediate true plus dry_run false, then resume the pause with approved true.
Enable the nightly trigger once the reported plan matches what you expect.

How to extend

Docker 28 renamed the prune parameter from keep-storage to reserved-space. On that daemon, bump api_version plus change the parameter name on reclaim_build_cache together.
Add an age floor with &filters={"until":{"24h":true}} on the prune call to keep anything used in the last day regardless of size.
Pinned cache needs the images gone first. Chain io.kestra.plugin.docker.cli.Prune with pruneType: IMAGES plus dangling: true after the verify when pinned_bytes is the thing holding the ceiling.
Send the plan to a dashboard instead of Slack by writing outputs.plan.values.p to your metrics store.
Run it per build host with a Loop over your runner list, one sub-execution each, so a floor can differ per host.

Guard the Docker BuildKit Build Cache with a Ceiling and a Most-Recently-Used Floor

Safety

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Guard the Docker BuildKit Build Cache with a Ceiling and a Most-Recently-Used Floor

Safety

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Couchbase TTL expiry triage sentinel

Create Linear Issues Automatically for Incident Management

CouchDB Active Task and Continuous Replication Sentinel

Create Jira Issues Automatically for Incident Management

CrateDB Shard Allocation and Cluster Reroute Sentinel

Cancel runaway Couchbase N1QL queries that starve the query service

New to Kestra?