Scheduled MinIO Bucket Retention Cleanup

Keep a self-hosted MinIO bucket from growing forever without deleting anything it shouldn't. This blueprint lists every object under a prefix on a nightly schedule, filters that list down to objects older than a configurable retention window, deletes only those expired keys one at a time, and reports how many objects and how many bytes were reclaimed. Nothing inside the retention window is ever touched, and a failed delete on one object never blocks the rest of the batch.

How it works

1. list_objects (io.kestra.plugin.minio.List) lists every object under the raw-exports/ prefix, recursively, filtering to files only. Each returned object carries its key, size, and lastModified timestamp.
2. find_expired_objects (io.kestra.plugin.scripts.python.Script) reads that listing from an inputFiles JSON file, computes a cutoff of retention_days (90 by default) before now, and keeps only the keys whose lastModified falls before the cutoff. It emits expired_keys, expired_count, and expired_size through Kestra.outputs.
3. delete_expired_objects (io.kestra.plugin.core.flow.Loop) iterates over expired_keys with a concurrencyLimit of 4, and for each one runs delete_object (io.kestra.plugin.minio.Delete) to remove that single key.
4. notify (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) reports the count of deleted objects and bytes freed. The errors block posts a separate Slack alert if listing or deleting fails.

Connection details (endpoint, accessKeyId, secretKeyId, bucket) are set on each io.kestra.plugin.minio task in the flow.

What you get

• An age-based retention sweep that only deletes what has actually expired, computed from real object timestamps rather than a fixed file count.
• Per-object deletes with bounded concurrency, so a single failure does not abort the whole cleanup run.
• A Slack report of exactly how many objects and how many bytes were reclaimed on every run.
• A pattern that works unmodified against any S3-compatible endpoint, not just AWS S3.

Who it's for

• Platform teams running self-hosted MinIO (or Ceph, SeaweedFS, Garage) as their primary object store and needing routine lifecycle cleanup.
• Data engineers managing landing zones or export buckets that accumulate files faster than anyone manually prunes them.
• Teams on air-gapped or on-prem infrastructure who cannot rely on a cloud provider's native bucket lifecycle rules.

Why orchestrate this with Kestra

MinIO does not ship a built-in, age-based lifecycle policy engine the way some managed clouds do, and even where lifecycle rules exist, they cannot branch on custom logic, report to Slack, or retry a failed delete independently of the rest of the batch. Kestra fills that gap: List and Delete give declarative building blocks, the Python step adds arbitrary filtering logic without a separate service, Loop isolates each delete so partial failures are visible and recoverable, and the whole policy lives in version-controlled YAML instead of a bucket-console setting nobody remembers configuring.

Prerequisites

• A running MinIO instance (or any S3-compatible object storage) reachable at the configured endpoint.
• A bucket named data-lake with a raw-exports/ prefix. Adjust both to match your environment.
• A Slack incoming webhook for notifications.

Secrets

• MINIO_ENDPOINT: URL to the MinIO endpoint, for example https://minio.internal:9000.
• MINIO_ACCESS_KEY_ID: MinIO access key.
• MINIO_SECRET_KEY_ID: MinIO secret key.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the four secrets above to your Kestra namespace.
2. Update bucket and prefix on list_objects to match the location you want to clean up.
3. Tune retention_days in find_expired_objects to match your actual retention policy.
4. Set disabled: false on the nightly trigger once you have confirmed the flow against a test bucket.

How to extend

• Turn retention_days into a flow input so different environments can apply different retention windows without editing YAML.
• Swap the fixed bucket and prefix for flow inputs to reuse the same flow across multiple buckets.
• Add a dry-run mode that logs expired_keys without calling delete_object, useful for validating a new retention policy before it deletes anything.
• Replace Delete with Copy to archive expired objects into a colder-storage bucket before removal instead of deleting them outright.

Links

• Plugin docs: https://kestra.io/plugins/plugin-minio