Glue ETL, Athena Validation, DynamoDB Cache

Stitch AWS Glue, Athena, and DynamoDB into a single governed data pipeline that runs an ETL job, validates its output, and publishes a ready-to-read metric. This blueprint solves a common gap in serverless analytics: Glue, Athena, and DynamoDB each have their own APIs and no shared scheduler, so chaining them reliably means writing glue code, polling for completion, and handling failures by hand. Here, Kestra coordinates the whole flow declaratively and only writes the KPI once the data has actually passed validation.

How it works

1. run_glue_job (io.kestra.plugin.aws.glue.StartJobRun) starts your Glue ETL job by name and, with wait: true, blocks until the job reaches a terminal state. A Glue failure stops the pipeline before any downstream task runs.
2. validate_with_athena (io.kestra.plugin.aws.athena.Query) runs a FETCH_ONE query against the curated table to read the headline metrics (row count and total amount) and confirm the ETL produced data.
3. write_kpi (io.kestra.plugin.aws.dynamodb.PutItem) caches the computed KPI in DynamoDB under a date-stamped kpi_id, capturing the Athena metrics and the Glue job state for fast application reads.
4. notify (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts the Glue state and row count to Slack.
5. A disabled io.kestra.plugin.core.trigger.Schedule trigger (cron: 0 4 * * *) is included for daily runs once you enable it.

What you get

• A serverless ETL-to-cache pipeline that runs Glue, Athena, and DynamoDB in the correct order.
• Validation built in: the KPI is only written after Athena confirms the curated data.
• Low-latency KPI reads from DynamoDB for dashboards and applications.
• Slack visibility on every run.

Who it's for

• Data engineers building serverless pipelines on AWS analytics services.
• Analytics teams that need a trusted daily KPI without standing up a database.
• Platform teams centralizing AWS workflow orchestration in Kestra.

Why orchestrate this with Kestra

Glue, Athena, and DynamoDB have no shared scheduler that can sequence a job, gate on its result, and write derived data in one place. Kestra fills that gap: event or schedule triggers start the run, retries and error handling keep it resilient, execution outputs give you lineage across the three services, and the whole pipeline is declarative YAML you can version and review. The wait: true gate ensures Athena and DynamoDB never run on a half-finished Glue job, something Glue's own job scheduling cannot enforce across services.

Prerequisites

An existing Glue job, an Athena database with a curated table, an S3 results location, and a DynamoDB table.

Secrets

• AWS_ACCESS_KEY_ID: AWS access key with Glue, Athena, S3, and DynamoDB permissions.
• AWS_SECRET_ACCESS_KEY: matching AWS secret access key.
• AWS_DEFAULT_REGION: AWS region for all three services.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL for run notifications.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Set the glue_job_name, athena_database, athena_output_location, and dynamodb_table inputs.
3. Adjust the Athena query in validate_with_athena to match your curated schema.
4. Run the flow once on demand, then enable the daily_pipeline schedule.

How to extend

• Swap the Slack notification for email or PagerDuty alerting.
• Add a second Athena query to write multiple KPIs in one run.
• Replace the schedule with a flow or webhook trigger to run on upstream data arrival.
• Add retries to run_glue_job for transient AWS failures.

Links

• Plugin: AWS
• Orchestrate AWS with Kestra