Convert a zipped CSV file to Parquet using a Python ETL workflow

Turn raw, compressed CSV drops into clean, analytics-ready Parquet files on a predictable schedule. This blueprint shows a complete ETL pattern in Kestra: pull a monthly ZIP archive from a public HTTPS endpoint, decompress it, and convert the extracted CSV to columnar Parquet using pandas and pyarrow inside a Docker task runner. It is a reusable template for any pipeline that lands compressed CSV data and needs to prepare it for lakehouses, warehouses, or DuckDB and Spark queries.

How it works

The flow computes a file_id variable from execution.startDate shifted back three months, formatted as yyyyMM. That id is reused across tasks so each run targets a different monthly archive.

1. get_zipfile uses io.kestra.plugin.core.http.Download to fetch https://divvy-tripdata.s3.amazonaws.com/{file_id}-divvy-tripdata.zip over HTTPS into Kestra's internal storage.
2. unzip uses io.kestra.plugin.compress.ArchiveDecompress with algorithm: ZIP to decompress the archive and expose the inner files via outputs.unzip.files.
3. parquet_output runs io.kestra.plugin.scripts.python.Script on the io.kestra.plugin.scripts.runner.docker.Docker task runner. It installs pandas and pyarrow as dependencies, reads {file_id}-divvy-tripdata.csv with pd.read_csv, writes {file_id}.parquet with df.to_parquet, and captures the result through outputFiles: ["*.parquet"].

What you get

• A clean monthly Parquet file in Kestra outputs, ready for downstream tasks.
• A self-contained Python environment per run, no shared venv drift.
• Reproducible runs: the same file_id math produces the same target archive every time.

Who it's for

• Data engineers standardizing landing-zone files into columnar formats.
• Analytics teams building monthly snapshots from public or partner CSV feeds.
• Platform teams replacing fragile cron plus bash glue with declarative flows.

Why orchestrate this with Kestra

Pandas and pyarrow have no scheduler, no retry policy, and no lineage. Kestra adds event and time triggers, automatic retries on flaky HTTP downloads, full execution history with inputs and outputs, and a declarative YAML definition you can version in Git. The Docker task runner isolates Python dependencies per execution, so upgrading pandas in one flow never breaks another.

Prerequisites

• A running Kestra instance with Docker available to the worker.
• Network access to divvy-tripdata.s3.amazonaws.com (or your own ZIP source).

Secrets

None. The source archive is public. If you adapt this flow to an authenticated endpoint, add the credentials as Kestra secrets and reference them with {{ secret('NAME') }}.

Quick start

1. Copy this blueprint into a namespace such as company.team.
2. Execute the flow once to confirm Docker pulls the Python image and writes the Parquet output.
3. Inspect the parquet_output task outputs to download or pass the file to a follow-up task.
4. Add a schedule trigger (for example 0 9 1 * *) to run on the first of every month.

How to extend

• Swap the URL for your own ZIP, GZIP, or TAR feed and adjust algorithm accordingly.
• Replace the inline script with io.kestra.plugin.scripts.python.Commands to run a versioned repo of transformations.
• Push the Parquet output to S3, GCS, or Azure Blob with the matching storage plugin.
• Register the file in a warehouse with io.kestra.plugin.jdbc.duckdb.Query or load it into BigQuery, Snowflake, or Postgres.
• Add data quality checks with Great Expectations or a SQL assertion task before publishing.

Links

• plugin-core
• plugin-compress
• plugin-script