Transform data from CSV files with Pandas in Python containers (in parallel)

Process many CSV files at once by fanning them out and running a dedicated Python and Pandas script for each file in its own isolated Docker container. This blueprint reads a list of CSV URLs, loops over them in parallel, and runs Pandas inside reproducible containers so a slow or failing file never blocks the rest. It is a clean pattern for parallel CSV processing, batch data transformation, and Python data pipelines where each input is independent.

How it works

• The csv task is an io.kestra.plugin.core.flow.ForEach loop that iterates over a list of CSV file URLs. With concurrencyLimit: 0, every value is processed at the same time with no cap on parallelism.
• For each value, the nested pandas task of type io.kestra.plugin.scripts.python.Script runs a Python script that reads the current file via pd.read_csv("{{ taskrun.value }}") and prints its schema and stats with df.info().
• Each script executes in a fresh container through the io.kestra.plugin.scripts.runner.docker.Docker task runner, with pandas declared under dependencies so the package is installed automatically.

What you get

• True parallel execution across all CSV inputs in one run.
• Full isolation per file: dependencies, memory, and failures are contained.
• Reproducible Python environments defined declaratively in YAML.
• A template you can point at your own CSV sources in seconds.

Who it's for

• Data engineers building batch ETL and data ingestion pipelines.
• Analysts and data scientists who process recurring sets of CSV exports.
• Platform teams standardizing reproducible Python jobs across a team.

Why orchestrate this with Kestra

A bare Python script can read one CSV at a time, but it cannot schedule itself, fan work out in parallel with isolation, retry a single failed file, or track lineage across runs. Kestra adds event and schedule triggers, per-task retries, full execution history and logs, and declarative YAML that captures the whole pipeline as code. ForEach gives you parallelism with a tunable concurrencyLimit, and the Docker task runner guarantees each file runs in a clean environment, filling the gap a standalone interpreter cannot.

Prerequisites

• A running Kestra instance.
• A Docker environment available to the worker for the Docker task runner.

Secrets

• None. This blueprint reads public CSV URLs and uses no {{ secret('NAME') }} values.

Quick start

1. Add this flow to a namespace in your Kestra instance.
2. Execute it as is to process the three sample CSV files in parallel.
3. Open each subtask in the Logs tab to inspect the df.info() output per file.
4. Replace the sample URLs under the csv task values with your own CSV locations.

How to extend

• Swap df.info() for real transformations: cleaning, aggregation, joins, or type casting.
• Write results to a database or object store with a follow-up task.
• Set a positive concurrencyLimit to throttle parallelism on large lists.
• Add a schedule or event trigger and per-task retry for production runs.
• Drive the values list dynamically from an upstream task instead of hardcoding URLs.

Links

• Python Script plugin docs