Pandas ETL - passing data between Python script tasks running in separate containers

Run a pandas ETL pipeline as two independent Python script tasks that still share files, using the io.kestra.plugin.core.flow.WorkingDirectory task to keep a common filesystem across Docker containers. Each Python step runs in its own isolated container with pinned pandas dependencies, while CSV artifacts produced by the extract step are picked up by the transform step and surfaced as a downloadable Kestra output. This pattern solves the classic problem of passing intermediate data between containerized Python tasks without bolting on S3, NFS, or custom volume mounts.

How it works

1. The parent etl task of type io.kestra.plugin.core.flow.WorkingDirectory creates a shared working directory and runs its children sequentially in that directory.
2. extract_csv is an io.kestra.plugin.scripts.python.Script running on the io.kestra.plugin.scripts.runner.docker.Docker task runner with pandas declared in dependencies. It builds a small DataFrame and writes raw_data.csv to the working directory.
3. transform_and_load_csv is a second io.kestra.plugin.scripts.python.Script Docker task that reads raw_data.csv, adds a computed Column4, and writes final.csv. It declares outputFiles: [final.csv] so Kestra pushes the file into internal storage as an execution output.
4. The resulting final.csv becomes available on the Outputs tab of the execution and can be referenced by downstream tasks or flows via Kestra's internal storage URIs.

What you get

• A reproducible, containerized pandas ETL with isolated Python environments per step.
• File sharing between Python tasks without external object storage.
• A downloadable final.csv artifact captured by Kestra's internal storage.
• A clean template for splitting extract, transform, and load into independent, restartable steps.

Who it's for

• Data engineers building Python ETL pipelines that need step isolation.
• Analytics engineers prototyping pandas transformations before promoting to warehouse SQL.
• Platform teams standardizing how Python scripts exchange intermediate files.

Why orchestrate this with Kestra

Plain python script.py or a cron job cannot give you per-step container isolation, typed inputs, retries on a single failing step, lineage across tasks, or downloadable outputs. Kestra adds event and schedule triggers, retries with backoff, audit logs, and declarative YAML so the pipeline is version controlled. The WorkingDirectory task fills the specific gap that pandas and Docker leave open: how to share files between containers without writing custom volume plumbing or pushing every intermediate to S3.

Prerequisites

• Kestra instance with the Docker task runner available (Docker daemon reachable from workers).
• Network access to pull the default Python image and install pandas.

Secrets

This flow does not use any {{ secret('NAME') }} values. Add secrets only if you extend the flow to read from a database or push to cloud storage.

Quick start

1. Copy this blueprint into a Kestra namespace such as company.team.
2. Confirm the Docker task runner can reach a Docker daemon on your workers.
3. Execute the flow from the UI.
4. Open the execution, go to the Outputs tab, and download final.csv.
5. Inspect the Logs tab to see the df.head() previews printed by each step.

How to extend

• Replace the inline dictionary in extract_csv with a real source: an API call, a SQL query, or a file pulled from S3 or GCS.
• Swap the in-memory DataFrame for a Polars or DuckDB pipeline by changing dependencies and the script body.
• Add a third step that loads final.csv into Postgres, BigQuery, or Snowflake using the matching Kestra plugin.
• Parameterize the schema and filenames with inputs so the same flow handles multiple datasets.
• Add a Schedule or Webhook trigger to run the ETL on a cadence or on demand from upstream systems.

Links

• Python scripts plugin: https://kestra.io/plugins/plugin-script-python
• Docker task runner plugin: https://kestra.io/plugins/core/docker-task-runner/io.kestra.plugin.scripts.runner.docker.docker
• Core flow plugin (WorkingDirectory): https://kestra.io/plugins/core