Scheduled MinIO Bucket Retention Cleanup

Keep a self-hosted MinIO bucket from growing forever without deleting anything it shouldn't. This blueprint lists every object under a prefix on a nightly schedule, filters that list down to objects older than a configurable retention window, deletes only those expired keys one at a time, and reports how many objects and how many bytes were reclaimed. Nothing inside the retention window is ever touched, and a failed delete on one object never blocks the rest of the batch.

How it works

1. list_objects (io.kestra.plugin.minio.List) lists every object under the raw-exports/ prefix, recursively, filtering to files only. Each returned object carries its key, size, and lastModified timestamp.
2. find_expired_objects (io.kestra.plugin.scripts.python.Script) reads that listing from an inputFiles JSON file, computes a cutoff of retention_days (90 by default) before now, and keeps only the keys whose lastModified falls before the cutoff. It emits expired_keys, expired_count, and expired_size through Kestra.outputs.
3. delete_expired_objects (io.kestra.plugin.core.flow.ForEach) iterates over expired_keys with a concurrencyLimit of 4, and for each one runs delete_object (io.kestra.plugin.minio.Delete) to remove that single key.
4. notify (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) reports the count of deleted objects and bytes freed. The errors block posts a separate Slack alert if listing or deleting fails.

Connection details (endpoint, accessKeyId, secretKeyId, bucket) are set once through pluginDefaults for the whole io.kestra.plugin.minio group, so every MinIO task in the flow shares the same credentials without repeating them.

What you get

• An age-based retention sweep that only deletes what has actually expired, computed from real object timestamps rather than a fixed file count.
• Per-object deletes with bounded concurrency, so a single failure does not abort the whole cleanup run.
• A Slack report of exactly how many objects and how many bytes were reclaimed on every run.
• A pattern that works unmodified against any S3-compatible endpoint, not just AWS S3.

Who it's for

• Platform teams running self-hosted MinIO (or Ceph, SeaweedFS, Garage) as their primary object store and needing routine lifecycle cleanup.
• Data engineers managing landing zones or export buckets that accumulate files faster than anyone manually prunes them.
• Teams on air-gapped or on-prem infrastructure who cannot rely on a cloud provider's native bucket lifecycle rules.

Why orchestrate this with Kestra

MinIO does not ship a built-in, age-based lifecycle policy engine the way some managed clouds do, and even where lifecycle rules exist, they cannot branch on custom logic, report to Slack, or retry a failed delete independently of the rest of the batch. Kestra fills that gap: List and Delete give declarative building blocks, the Python step adds arbitrary filtering logic without a separate service, ForEach isolates each delete so partial failures are visible and recoverable, and the whole policy lives in version-controlled YAML instead of a bucket-console setting nobody remembers configuring.

Prerequisites

• A running MinIO instance (or any S3-compatible object storage) reachable at the configured endpoint.
• A bucket named data-lake with a raw-exports/ prefix. Adjust both to match your environment.
• A Slack incoming webhook for notifications.

Secrets

• MINIO_ENDPOINT: URL to the MinIO endpoint, for example https://minio.internal:9000.
• MINIO_ACCESS_KEY_ID: MinIO access key.
• MINIO_SECRET_KEY_ID: MinIO secret key.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the four secrets above to your Kestra namespace.
2. Update bucket and prefix on list_objects to match the location you want to clean up.
3. Tune retention_days in find_expired_objects to match your actual retention policy.
4. Set disabled: false on the nightly trigger once you have confirmed the flow against a test bucket.

How to extend

• Turn retention_days into a flow input so different environments can apply different retention windows without editing YAML.
• Swap the fixed bucket and prefix for flow inputs to reuse the same flow across multiple buckets.
• Add a dry-run mode that logs expired_keys without calling delete_object, useful for validating a new retention policy before it deletes anything.
• Replace Delete with Copy to archive expired objects into a colder-storage bucket before removal instead of deleting them outright.

Links

• Plugin docs: https://kestra.io/plugins/plugin-minio