Extract a CSV file via HTTP API and upload it to S3 by using scheduled

Automate hourly ingestion from any public HTTP endpoint into Amazon S3. This Kestra blueprint downloads a ZIP archive, extracts a CSV from it, and uploads the file to an S3 bucket under a date and time partitioned key, giving you a reproducible, append-only landing zone for downstream pipelines (Snowflake, Athena, DuckDB, dbt, Spark, Glue). It is the missing scheduler and orchestrator between an external data source and your S3 data lake: no cron jobs to babysit, no Lambda glue code, no ad hoc EC2 scripts.

How it works

1. A io.kestra.plugin.core.trigger.Schedule trigger fires every hour using cron @hourly.
2. get_zip_file runs io.kestra.plugin.core.http.Download to fetch the source ZIP archive over HTTPS. By default it pulls the Global Power Plant Database sample, but the URI is a one-line change.
3. unzip runs io.kestra.plugin.compress.ArchiveDecompress with algorithm: ZIP against the downloaded file, exposing each extracted file as an internal storage URI on outputs.unzip.files.
4. csv_upload runs io.kestra.plugin.aws.s3.Upload and pushes global_power_plant_database.csv to the bucket from inputs.bucket, using the key powerplant/{{ trigger.date ?? execution.startDate | date('yyyy_MM_dd__HH_mm_ss') }}.csv so every run lands in its own object.

What you get

• Hourly, hands-off ingestion from any HTTP source into S3.
• Time-partitioned object keys (yyyy_MM_dd__HH_mm_ss) that are safe for backfills and replays.
• Internal storage handoff between tasks (no local disk juggling).
• A repeatable pattern you can fork for any public CSV, JSON, or ZIP feed.

Who it's for

• Data engineers building a raw landing zone in S3.
• Analytics teams ingesting open data, vendor exports, or partner feeds.
• Platform teams replacing brittle cron + bash + aws-cli pipelines with declarative YAML.

Why orchestrate this with Kestra

Amazon S3 has no scheduler, no retry engine, and no lineage. AWS EventBridge plus Lambda can trigger jobs, but you still end up writing and maintaining Python or Node glue, IAM roles, and CloudWatch wiring. Kestra gives you:

• A native Schedule trigger and event-based triggers in one place.
• Automatic retries, timeouts, and error branches per task.
• Execution-level lineage and logs across HTTP, decompression, and S3 steps.
• Declarative YAML you can version in Git and review like code.
• A single orchestration plane for the rest of your stack (Snowflake, BigQuery, dbt, Airbyte, Kubernetes).

Prerequisites

• An AWS account and an S3 bucket you can write to.
• The bucket name passed as the bucket input (defaults to declarative-data-orchestration).
• AWS credentials available to the Kestra worker (via instance profile, environment, or pluginDefaults for io.kestra.plugin.aws).

Secrets

This blueprint does not reference Kestra secret() values directly. For production, wire your AWS credentials through pluginDefaults or environment variables, for example AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and keep them out of the flow YAML.

Quick start

1. Copy this flow into a namespace such as company.team.
2. Set the bucket input default to your S3 bucket.
3. Configure AWS credentials for the io.kestra.plugin.aws.s3.Upload task (instance profile or pluginDefaults).
4. Save and execute once manually to confirm the object lands at s3://<bucket>/powerplant/<timestamp>.csv.
5. Leave the hourly schedule on, or change the cron to fit your refresh cadence.

How to extend

• Swap the uri on get_zip_file for any public or authenticated HTTP endpoint (add headers for tokens).
• Replace ArchiveDecompress with io.kestra.plugin.compress.ArchiveExtract for TAR or GZIP sources.
• Add a io.kestra.plugin.aws.s3.List or Copy step to fan out to multiple buckets or regions.
• Chain a Snowflake COPY INTO, Athena query, or dbt run right after the upload.
• Switch the trigger to io.kestra.plugin.core.trigger.Webhook to ingest on demand instead of on a clock.

Links

• plugin-aws docs
• AWS orchestration with Kestra