Ingest data to AWS S3 with Git, Python, Apache Iceberg, AWS Glue and Amazon Athena

This blueprint automates incremental data ingestion into an AWS S3 data lakehouse built on Apache Iceberg, AWS Glue, and Amazon Athena. It solves a common ETL problem: detecting newly landed raw files, transforming them with versioned Python code from Git, merging them into Iceberg tables, and compacting the result, all without manual intervention or a standing data pipeline service. The flow keeps your transformation logic reproducible by cloning it fresh from a Git repository on every run.

How it works

1. list_objects (io.kestra.plugin.aws.s3.List) scans the configured bucket and prefix for new raw CSV files.
2. check (io.kestra.plugin.core.flow.If) evaluates outputs.list_objects.objects and only proceeds when new files exist.
3. process_new_objects (io.kestra.plugin.core.flow.WorkingDirectory) wraps the transformation step. Inside it, clone_repository (io.kestra.plugin.git.Clone) pulls the ETL scripts from GitHub, then ingest_to_datalake (io.kestra.plugin.scripts.python.Commands) runs etl/aws_iceberg_fruit.py in a Docker taskRunner with awswrangler and kestra installed, writing cleaned data into the Iceberg lake.
4. merge_query (io.kestra.plugin.aws.athena.Query) runs a MERGE INTO statement to upsert raw rows into the target Iceberg table.
5. optimize (io.kestra.plugin.aws.athena.Query) runs OPTIMIZE ... REWRITE DATA USING BIN_PACK to compact small files.
6. move_to_archive (io.kestra.plugin.aws.cli.AwsCLI) moves processed files to an archive/ prefix so they are not reprocessed.

A Schedule trigger (@hourly, shipped disabled) drives the cadence.

What you get

• Incremental, idempotent ingestion that skips runs when no new files exist.
• Iceberg upserts via Athena MERGE INTO, with automatic table compaction.
• Versioned transformation logic pulled from Git on every run.
• Reproducible Python execution in an isolated Docker container.
• Automatic archival of processed source files.

Who it's for

• Data engineers building or maintaining an S3 lakehouse.
• Analytics teams standardizing raw-to-curated ingestion on AWS.
• Platform teams who want Git-versioned, auditable ETL.

Why orchestrate this with Kestra

AWS scheduling primitives like EventBridge or Glue triggers can start a job, but they do not give you a single declarative pipeline that conditionally branches on detected files, clones versioned code, runs containerized Python, and chains Athena queries with built-in retries and full execution lineage. Kestra expresses the whole flow as readable YAML, captures inputs, outputs, and logs per task, and lets you swap the disabled schedule for event triggers or webhook triggers without rewriting the logic.

Prerequisites

• An AWS account with S3, Glue, and Athena configured for Iceberg.
• A Git repository containing your ETL scripts (defaults to github.com/kestra-io/scripts).
• Docker available to the Kestra worker for the Python task runner.

Secrets

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION

Quick start

1. Add the three AWS secrets above to your Kestra instance.
2. Set the bucket, prefix, and database variables to match your environment.
3. Point clone_repository at your own scripts repo and command if needed.
4. Run the flow once manually, then enable the hourly_schedule trigger.

How to extend

• Replace the Schedule trigger with an S3 event or webhook trigger for near-real-time ingestion.
• Parameterize the bucket, prefix, and table names with flow inputs.
• Add data-quality checks or a Slack notification after merge_query.
• Swap the Python script for dbt, Spark, or another transformation engine.

Links

• plugin-aws documentation
• AWS orchestration with Kestra