Process files in parallel

Process a dynamic set of files concurrently in a single Kestra workflow. This blueprint generates multiple files in a shell task, captures them in Kestra internal storage, then fans the list out so each file is handled in its own parallel task run. It solves a common data engineering problem: when an upstream step emits many files, you want to process them at the same time instead of looping sequentially, without writing custom threading or queue code.

How it works

1. The bash task (io.kestra.plugin.scripts.shell.Commands) runs on the io.kestra.plugin.core.runner.Process task runner. It creates an out directory and writes four text files, then declares outputFiles: out/** so all generated files are persisted to Kestra internal storage as task outputs.
2. The each task (io.kestra.plugin.core.flow.ForEach) reads outputs.bash.outputFiles and iterates over the list using the Pebble expression {{ outputs.bash.outputFiles | jq('.[]') }}. With concurrencyLimit: 0, every value runs in parallel with no cap.
3. For each file, two child tasks run: path (io.kestra.plugin.core.debug.Return) echoes the current {{ taskrun.value }}, and contents (io.kestra.plugin.scripts.shell.Commands) runs cat on that file path to read its contents.

With four files and two tasks each, the flow produces eight parallel task runs.

What you get

• Dynamic parallelism that scales to however many files the upstream task emits.
• Files persisted in internal storage and passed downstream by reference.
• A clean fan-out pattern using only core plugins, no extra dependencies.
• Per-file isolation, so one failing file does not block the others.

Who it's for

• Data engineers building file ingestion and transformation pipelines.
• Platform teams replacing ad hoc shell loops with declarative orchestration.
• Anyone who needs to process a variable batch of files concurrently.

Why orchestrate this with Kestra

A standalone shell script can loop over files, but it cannot give you parallel execution with backpressure, per-item retries, event triggers, or lineage out of the box. With Kestra you get declarative YAML, automatic capture of outputFiles into internal storage, full execution lineage across every parallel task run, retries scoped per task, and the option to start the flow from event triggers (file arrival, webhook, schedule) rather than a cron line in a crontab the underlying shell has no concept of.

Prerequisites

• A running Kestra instance.
• A worker able to run the Process task runner (shell available on the host).

Secrets

This flow references no secrets. It uses only local shell commands and Kestra internal storage.

Quick start

1. Add this flow to a Kestra namespace.
2. Trigger an execution manually from the UI.
3. Open the each task in the Gantt and Topology views to watch the parallel task runs.
4. Inspect the contents task logs to confirm each file was read.

How to extend

• Swap the bash task for a Python, R, or Node.js script that produces the files.
• Set a non-zero concurrencyLimit to throttle how many files process at once.
• Replace the contents task with real transformation, validation, or upload logic.
• Add an io.kestra.plugin.core.trigger.Schedule or a file-detection trigger to run on new data automatically.

Links

• Plugin docs: plugin-scripts
• Plugin docs: plugin-core