Fan Out S3 Object Events to Lambda and SNS

Watch an Amazon S3 bucket and turn every new object into routed work. This event-driven blueprint detects files as they land in S3, inspects each object key, and fans them out by type: JSON payloads go to an AWS Lambda function for processing, while all other files are announced on an SNS topic. A Slack message closes the loop so your team sees the outcome in real time. It solves the common problem of reacting to data drops, uploads, and integration handoffs without standing up bespoke event plumbing or polling scripts.

How it works

1. The io.kestra.plugin.aws.s3.Trigger polls the incoming/ prefix in the configured bucket every PT1M and fires once per new object (maxKeys: 1, action: NONE so files stay in place).
2. An io.kestra.plugin.core.flow.If task evaluates the object key with a jq expression that checks whether it ends in .json.
3. If true, io.kestra.plugin.aws.lambda.Invoke calls your Lambda (functionArn) with the object key and uri as the payload.
4. Otherwise, io.kestra.plugin.aws.sns.Publish posts a message to your SNS topic (topicArn) announcing the new object.
5. io.kestra.plugin.slack.notifications.SlackIncomingWebhook posts the routing outcome to Slack.

What you get

• Real-time, per-object reactions to files landing in S3.
• Type-based routing in a single declarative flow.
• Serverless processing through Lambda for structured JSON.
• Broadcast fan-out via SNS for everything else.
• Slack visibility on each run.

Who it's for

• Data engineers building ingestion and landing-zone pipelines.
• Platform teams wiring up event-driven AWS architectures.
• DevOps engineers replacing ad hoc S3 event glue with orchestrated flows.

Why orchestrate this with Kestra

S3 event notifications can fire Lambda or SNS directly, but that native wiring stops at the first hop: it cannot branch on content, chain downstream steps, retry a failed Lambda call, or give you a unified execution history. Kestra adds an event Trigger, declarative YAML branching, automatic retries, full run lineage across Lambda and SNS, and one place to observe every object that flowed through. You get the routing logic and the audit trail without bolting EventBridge rules and dead-letter queues together by hand.

Prerequisites

• An AWS account with an existing Lambda function and SNS topic.
• An S3 bucket and prefix to watch.
• A Slack incoming webhook.

Secrets

• AWS_ACCESS_KEY_ID: AWS access key with S3, Lambda, and SNS permissions.
• AWS_SECRET_ACCESS_KEY: matching AWS secret key.
• AWS_DEFAULT_REGION: AWS region for the resources.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the secrets above in your Kestra namespace.
2. Point the object_landed trigger at your bucket and prefix.
3. Pass lambda_function_arn and sns_topic_arn as inputs.
4. Drop a .json file and a non-JSON file into the prefix to see both branches route.

How to extend

• Add more If branches (or a Switch) to fan out across additional file types or prefixes.
• Switch the trigger action to MOVE or DELETE to avoid reprocessing handled objects.
• Replace the Slack step with email, PagerDuty, or a downstream subflow.
• Have the Lambda branch write results back to S3 or a data warehouse.

Links

• Plugin docs: https://kestra.io/plugins/plugin-aws
• Orchestrate AWS with Kestra: https://kestra.io/orchestration/aws