Kafka moves the events. Kestra turns them into work: one execution per record in real time, or batched pulls on an interval, with the consumer, retries, and downstream chain handled for you. Stream into the warehouse, produce events after a pipeline, and trace every record in one history.
Reacting to Kafka usually means running and babysitting your own consumer service. Kestra makes the topic a trigger: RealtimeTrigger starts one execution per record, Trigger batches records on an interval, and Consume reads mid-flow. From there it chains straight into Snowflake or BigQuery loads, dbt transforms, and notifications, with retries and one execution history per record. No consumer infrastructure to operate.
Kafka owns the log, the partitions, and the offsets. Kestra owns what happens to each record: how it triggers a flow, what runs next, who is alerted, and where the per-record audit trail lives.
The io.kestra.plugin.kafka.RealtimeTrigger consumes records as they arrive and starts one execution per record, with manual offset commits and configurable deserializers. No long-running consumer to write, deploy, and monitor; the trigger is the consumer, and each record gets its own traced run.
When per-record is too granular, the Trigger polls on a fixed interval (default one minute), batches records into one execution, and stores them in internal storage. Use it for micro-batch loads into the warehouse where grouping beats one-at-a-time.
Consume writes fetched records to internal storage as ION, ready to load into Snowflake or BigQuery. A flow can consume a topic, transform with dbt, and land curated rows, turning a raw stream into warehouse tables with retries on each step.
Orchestration is not only consuming. Produce publishes records to a topic with configurable serializers and transactional sends, so a flow can emit a completion event, a CDC-style change, or a downstream signal once its work is done. Other consumers and Kestra triggers pick it up.
Modern Kafka supports share groups for queue-style consumption with explicit acknowledgements. Kestra exposes this through groupType: SHARE on the consume and trigger tasks, so a topic can be worked as a queue with multiple competing consumers, not just a partitioned log.
The usual cost of reacting to Kafka is a fleet of consumer apps to deploy, scale, and page on. Kestra runs the consumer inside the orchestrator, so reacting to a topic is a trigger definition, not a service. Offsets, retries, and execution history come built in.
Patterns streaming and platform teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
The RealtimeTrigger starts one execution per record. Kestra routes by payload, runs the matching work, and alerts on anomalies. Each record gets its own traced run, with no consumer service to operate.
RealtimeTrigger fans each record into its own traced run.
Offsets commit after the work succeeds, so failures replay.
A Switch sends each record type down its own path.
The batch Trigger pulls records on an interval, Kestra transforms them, and loads into Snowflake or BigQuery. Micro-batching turns a raw stream into curated warehouse tables with per-step retries.
The Trigger groups records into one execution per pull.
Consumed records land as ION, ready to load.
A failed load retries without re-consuming the batch.
After a flow completes its work, Produce publishes a completion or CDC-style event to a topic. Downstream consumers and other Kestra flows react to it, decoupling the producer from everyone who cares.
Consumers react to the event without coupling to the producer.
Produce uses transactional sends by default.
A Kestra trigger on the topic chains the next pipeline.
With groupType: SHARE, multiple competing consumers pull from one topic with explicit acks, queue-style. Kestra runs the worker logic per record and acks on success, so a backlog drains across parallel executions.
Share groups give competing-consumer behavior with explicit acks.
A backlog spreads across parallel executions.
Records ack only after the work completes.
Kestra consumes a Kafka topic and writes to AWS S3, a database, or an HTTP API, bridging Kafka to systems it does not reach natively. One execution history covers the consume and the downstream write.
Land records in S3, a database, or an API in the same flow.
The consume and the downstream write share one execution ID.
A failed upload retries without re-consuming the records.
Kestra gives us a modern orchestration platform we can run ourselves without compromising on governance.
| Capability | | Kafka Connect | | |
|---|---|---|---|---|
| One execution per record | RealtimeTrigger | Sink/source only, no logic | Workflow per signal, coded | Custom consumer code |
| Batched interval pulls | Trigger task | Connector config | Coded | Custom code |
| Chain consume with dbt and warehouse | Native plugins | Sink connectors only | Code activities | Python code required |
| Produce events from a flow | Produce task | Source connector | Coded activity | Custom code |
| Share-group queue semantics | groupType SHARE | No | Not applicable | Custom code |
| Per-record execution history | One run per record | Connector logs | Workflow history | App logs |
| No consumer service to operate | Trigger is the consumer | Connect cluster to run | Worker fleet to run | Service to run |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Trigger per record or on batched pulls, stream into the warehouse, produce events after a pipeline, all with no consumer service to operate. Open source, self-hosted, event-driven.
