Process partitions in parallel in Python, report outputs and metrics

Process a list of data partitions in parallel using isolated Python scripts, each running in its own Docker container, and emit custom metrics for every partition. This pattern solves a common data engineering problem: when a dataset is split into many files (Parquet, CSV, JSON), you want to process them concurrently rather than one at a time, while still tracking how many rows each file held and how long each unit of work took. By emitting Kestra counters and timers per partition, you turn an opaque batch job into an observable, measurable pipeline.

How it works

1. The get_partitions task is an io.kestra.plugin.scripts.python.Script running on io.kestra.plugin.scripts.runner.docker.Docker. It builds a list of partition filenames and publishes them as a flow output with Kestra.outputs({'partitions': partitions}).
2. The process_partitions task is an io.kestra.plugin.core.flow.ForEach iterating over {{ outputs.get_partitions.vars.partitions }} with concurrencyLimit: 0, so every partition runs at the same time.
3. Inside the loop, the partition task is another Python Script in Docker. It reads {{ taskrun.value }} to know which file it owns, simulates processing, then emits Kestra.counter('nr_rows', ...) and Kestra.timer('processing_time', ...), each tagged with the partition name.

What you get

• Parallel execution of all partitions, each fully isolated in its own container.
• Per-partition metrics (nr_rows counter and processing_time timer) visible in the Metrics tab.
• A dynamic fan-out driven entirely by an upstream task output, with no hardcoded loop.
• A reusable template for any split-file or chunked workload.

Who it's for

• Data engineers running partitioned batch jobs over many files.
• Python developers who want concurrency without managing a thread pool or executor.
• Platform teams that need per-unit observability across a fan-out.

Why orchestrate this with Kestra

A bare Python script can loop over files, but it cannot natively retry a single failed partition, cap concurrency declaratively, surface per-partition metrics, or expose execution lineage. Kestra adds event triggers, automatic retries, declarative YAML for the whole fan-out, and built-in metric tracking. The ForEach task handles parallelism and concurrency limits for you, and each partition becomes an independently observable task run rather than a hidden iteration inside one process.

Prerequisites

• A running Kestra instance with Docker available to the task runner.

Secrets

• None. This flow references no {{ secret('NAME') }} values.

Quick start

1. Add this flow to a namespace in your Kestra instance.
2. Execute it from the UI.
3. Watch the process_partitions task fan out into parallel partition runs.
4. Open the Metrics tab to inspect nr_rows and processing_time per partition.

How to extend

• Replace get_partitions with a real listing task (object storage, a database query, or a directory scan).
• Swap the simulated work in partition for actual extraction, transformation, or loading logic.
• Set a non-zero concurrencyLimit to throttle parallelism when downstream systems are rate sensitive.
• Add a final task to aggregate per-partition outputs into a single report.

Links

• Python plugin documentation