Trigger S3, Lambda, Glue, EMR, DynamoDB, and Athena from one flow. S3 events fan out without EventBridge rules, EMR Serverless dispatches on backlog, Lambda chains into Glue and Athena, cross-account ops run through stsRole, all in one execution history.
AWS gives you the services. Kestra wires them together without EventBridge rule sprawl, without Step Functions JSON, and without Lambda functions whose only purpose is to call the next Lambda. The IAM model, the data, and the compute all stay in your account.
s3.Trigger polls a bucket on an interval, applies prefix and regex filters, downloads matched objects to Kestra internal storage, and fires the flow once per file. Optional move or delete after processing prevents reprocessing. The EventBridge to Lambda to Step Functions chain collapses into one trigger and one flow, with the routing logic readable in the flow YAML.
lambda.Invoke calls a Lambda by ARN or name with a JSON payload. Trigger from Kafka, Slack, dbt completion, ServiceNow webhook, anything Lambda's native triggers cannot reach without a glue function. The response body is stored in internal storage and the task fails on functionError. Pair with retry policies for transient throttling.
emr.CreateClusterAndSubmitSteps launches a transient cluster, runs initial steps, and optionally waits for terminal state. emr.SubmitSteps appends jobs to a running cluster. emr.DeleteCluster tears it down. EMR Serverless covers the steady-state side with CreateServerlessApplicationAndStartJob. Backlog-driven autoscale becomes a flow: SQS depth threshold breaches, the flow spins a Spark job.
Every AWS task accepts stsRoleArn, stsRoleSessionName, stsRoleExternalId. Chain ops across accounts in one flow: drop a file in the dev-account S3 bucket, run a Glue job in the staging-account, write the output to a prod-account DynamoDB table. No federation setup, no separate flows per account, no shared long-lived credentials.
athena.Query runs SQL and fetches results to internal storage. glue.StartJobRun, GetJobRun, StopJobRun handles Glue lifecycle. dynamodb.GetItem, PutItem, Query, Scan, DeleteItem are first-class operations. Chain Athena to dbt to Snowflake to Slack without Step Functions JSON or three separate orchestrators.
eventbridge.PutEvents writes events to a custom bus when a flow step finishes. Other consumers (downstream Lambdas, third-party SaaS, on-call paging) get notified. The flow does not depend on EventBridge rules for its own routing, routing lives in the flow YAML. Decouple the publish from the orchestration.
Patterns engineering teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
s3.Trigger polls the bucket. Files under /raw/orders/ route to a dbt build that transforms the order data. Files under /raw/clicks/ route to a Lambda that cleanses the clickstream and forwards it to Kinesis. Both paths emit an SNS message when complete and notify Slack. The EventBridge plus two Lambdas plus a Step Function pattern collapses into one flow definition.
Prefix-driven routing in the flow, not in an EventBridge rule. Adding a third prefix path is a YAML edit, not a CloudFormation stack update.
s3.Trigger optionally moves or deletes processed files. The next poll skips them automatically.
One execution links the original file, the routing decision, the downstream task output, the SNS message, and the Slack notification.
A scheduled flow runs every 5 minutes. It reads the SQS queue depth from CloudWatch and the active EMR Serverless job count. If the queue is above a threshold and no job is running, emr.CreateServerlessApplicationAndStartJob spins a Spark job that drains the backlog. emr.GetJobRun polls until SUCCEEDED. Below threshold, the flow exits. Backlog-driven autoscale without DataDog watcher plus Lambda controller plus retry logic.
The flow spins compute only when the queue actually has work. Quiet weekends do not spin idle clusters.
EMR Serverless for steady-state batches. emr.CreateClusterAndSubmitSteps for one-off heavy runs that need custom node types.
The flow checks for an active job before starting a new one. No accidental parallel runs from overlapping schedule fires.
A weekly flow iterates a list of AWS accounts. Per account, the flow assumes a rotation role via stsRoleArn, creates a new access key with cli.AwsCLI, writes the new key to AWS Secrets Manager in the target account, and marks the old key inactive. The flow Pauses before reaching the prod account. A named reviewer resumes, the prod rotation runs, and the old key is scheduled for deletion 24 hours later by a follow-up flow.
Each iteration assumes a different stsRoleArn. No long-lived credentials in the flow, no federation tax.
The Pause task records who approved the prod rotation and when. The audit log links to the execution.
A follow-up flow scheduled at execution time deletes the old keys, after the rotation has been verified.
glue.StartJobRun kicks off the daily ETL job that lands curated data in the analytics S3 bucket. glue.GetJobRun polls every minute until SUCCEEDED. athena.Query runs a validation SQL against the output and returns the row count. An If task gates: if row count is within the expected range, dynamodb.PutItem writes the daily aggregate to the metrics table. Slack notifies with the row count and the bucket URI.
Glue, Athena, DynamoDB all chained directly. No Step Functions, no Glue Workflows, no glue Lambda.
If the Athena row count is outside the expected window, DynamoDB is not updated and on-call gets paged.
Each task carries its own retry policy. A Glue throttling error retries the Glue step only, not the whole flow.
Kestra was the only tool that combined true multi-tenant isolation, metadata-driven orchestration, and easy integration with our existing AWS and Databricks environments. It provided the foundation we needed to scale confidently.
One blueprint per use case above. Copy the YAML, set your AWS credentials in Kestra secrets, point at your resources, ship it.
s3.Trigger polls the bucket. An If task routes /raw/orders/ to dbt build and /raw/clicks/ to a Lambda. Both paths publish to SNS on completion and notify Slack.
Every 5 minutes, the flow reads SQS queue depth via CloudWatch. If above threshold, it starts an EMR Serverless Spark job and polls until SUCCEEDED. Below threshold, the flow exits. No idle compute on quiet weekends.
Starts a Glue job, polls until SUCCEEDED, runs an Athena validation query, gates a DynamoDB write on the row count. Failed validation pages on-call instead of writing bad data.
| Capability | | | EventBridge rules + Lambda fan-out | Custom code on EC2 / ECS |
|---|---|---|---|---|
| S3 trigger with prefix/regex filter and fan-out | s3.Trigger with action MOVE or DELETE on processed files | S3 notification to Lambda to Step Function, three resources | S3 to EventBridge to rule to Lambda, four resources | Custom poller code |
| Lambda dispatch from non-AWS event sources | lambda.Invoke from any Kestra trigger | API Gateway + Lambda + Step Function entry | Pub/Sub or webhook to EventBridge to Lambda | Custom HTTP listener |
| Approval gate before a service call | Pause + resume, named reviewer logged | Step Function Wait + custom approval Lambda | Not native | Custom approval UI |
| EMR Serverless on backlog signal | CloudWatch query + If + StartServerlessJobRun in one flow | Step Function with custom watcher Lambda | EventBridge schedule + Lambda controller | Custom watcher service |
| Cross-account ops via stsRole on every task | stsRoleArn property on each AWS task | Custom role-assume Lambda step | Cross-account EventBridge bus + per-account rules | Custom assume-role code |
| Glue + Athena + DynamoDB in one flow | Native plugin tasks, outputs flow forward | Step Function with three Lambdas | Three EventBridge rules + glue logic | Custom code |
| Chain AWS with non-AWS tools (dbt, Slack, GitHub) | Outputs flow into any plugin task | Lambda glue for each external call | EventBridge to API Destinations + custom logic | Custom integrations |
| Self-hosted, air-gapped, OSS edition | Self-hosted by default, OSS edition free | AWS-managed only | AWS-managed only | Self-hosted |
| Vendor lock-in | AWS plugin is one of 1300+; portable orchestration | Step Functions ASL is AWS-only | EventBridge rules are AWS-only | Custom code, portable but unsupported |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Automate infrastructure workflows in Kestra
How Kestra orchestrates AWS, Kubernetes, Terraform, Cloudflare, and the rest of the stack in one declarative flow.
Run AWS workloads alongside Kubernetes and GitHub Actions
The common pattern: GitHub Actions builds and pushes to ECR, Kestra dispatches an EMR Serverless job from the new image, results land in DynamoDB. One execution across three platforms.
AWS plugin reference
Every AWS task (S3, Lambda, Glue, EMR, Athena, DynamoDB, SQS, SNS, EventBridge, CloudWatch, ECR, CloudFormation, CLI) with copy-ready YAML and full property reference.
Fan out S3 events to dbt and Lambda from one trigger. Spin EMR Serverless on backlog. Rotate IAM keys across accounts with a prod gate. Chain Glue, Athena, and DynamoDB in one execution history. Open source, self-hosted, event-driven.
