Real-Time Kafka Record Processing with Per-Record Executions

Turn a Kafka topic into per-record workflow executions with event-driven orchestration. This blueprint consumes each message from a Kafka stream the moment it arrives, starts one Kestra execution per record, branches on the message payload, and routes anomalies to Slack while logging normal records. It solves a common streaming problem: reacting to individual events in real time without building, deploying, and babysitting a standalone consumer service. The trigger is the consumer, so offsets, retries, and per-record execution history are managed for you.

How it works

• The on_record trigger (io.kestra.plugin.kafka.RealtimeTrigger) subscribes to the transactions topic, deserializes each message as JSON, and launches one execution per record using the kestra-realtime-processor consumer group.
• The route_record task (io.kestra.plugin.core.flow.If) inspects the record with a jq expression on trigger.value and branches when .status equals FAILED.
• On a failed record, alert_anomaly (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts a message to Slack with the topic, offset, and execution id.
• Otherwise, log_record (io.kestra.plugin.core.log.Log) records that the message was processed normally with its partition and offset.

What you get

• Per-record executions with full input, output, and log history.
• Conditional routing so anomalies are surfaced immediately.
• Slack alerting on failures, out of the box.
• No consumer application to write, package, or operate.

Who it's for

• Data and platform engineers building event-driven pipelines on Kafka.
• Teams that need real-time anomaly detection or alerting on a stream.
• Anyone who wants per-event observability without standing up a custom consumer.

Why orchestrate this with Kestra

Kafka delivers messages, but it has no scheduler and no workflow engine of its own. Kestra adds the event trigger that turns each record into a fully observable execution, with automatic retries, replay on failure, and lineage across every task. Everything is declarative YAML, version controlled and reviewable, instead of bespoke consumer code you have to maintain. The gap Kafka itself cannot fill is orchestration: branching, downstream actions, alerting, and per-event audit history.

Prerequisites

• A Kafka topic (transactions) producing JSON messages with a status field.

Secrets

• KAFKA_BOOTSTRAP_SERVERS: Kafka bootstrap servers, e.g. broker1:9092,broker2:9092
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Set the secrets above in your Kestra namespace.
2. Point the trigger topic at your stream and set the groupId.
3. Adjust the If condition and jq expression to match your message shape.
4. Set disabled: false on the trigger to start consuming.

How to extend

• Add tasks under the then branch to quarantine, enrich, or remediate failed records.
• Swap the Slack alert for PagerDuty, email, or a downstream subflow.
• For competing consumers, set groupType: SHARE on the trigger and acknowledge with acknowledgeType.
• For high-throughput streams where per-record executions are too granular, use the batch Kafka Trigger for grouped micro-batches.

Links

• Kafka plugin docs
• Orchestrate Kafka with Kestra