Extract data from a REST API and send it to a Kafka topic using the Kafka

Move data from an HTTP REST API into Apache Kafka without writing a producer service. This flow calls a REST endpoint, captures the JSON response, and publishes it as a message to a Kafka topic, giving you a repeatable, observable bridge between request-response APIs and your event-streaming backbone. It solves the common problem of getting external or internal API data onto a Kafka topic so downstream consumers, stream processors, and sinks can react to it in real time.

How it works

1. The api task (io.kestra.plugin.core.http.Request) sends an HTTP request to https://dummyjson.com/products and stores the response body.
2. The produce task (io.kestra.plugin.kafka.Produce) builds a message from a from map containing key, value, timestamp, and headers. The value is set to {{ outputs.api.body }}, the JSON body returned by the api task, and the timestamp uses {{ execution.startDate }}.
3. The message is serialized with keySerializer: STRING and valueSerializer: JSON, then written to the topic mytopic on the cluster defined by properties.bootstrap.servers.

What you get

• A working pattern for turning any REST API response into a Kafka message.
• Custom message keys, headers, and timestamps that you control per record.
• JSON serialization wired to a live API payload, ready to adapt to your own schema.
• Full execution logs, outputs, and run history for every publish.

Who it's for

• Data engineers building ingestion paths into Kafka.
• Platform teams bridging external APIs to internal event streams.
• Developers prototyping streaming pipelines before standing up a dedicated producer.

Why orchestrate this with Kestra

Kafka producers run inside application code, but Kestra makes producing messages a first-class, scheduled, and event-driven workflow. You get declarative YAML instead of bespoke producer services, automatic retries on transient API or broker failures, and end-to-end lineage from the upstream API call to the published record. Kafka's own tooling has no scheduler or dependency engine: it cannot fetch the API, sequence the request before the publish, or retry the whole pipeline as a unit. Kestra fills that gap with triggers, conditional logic, and full observability.

Prerequisites

• A running Kafka cluster reachable at the address in properties.bootstrap.servers.
• A topic named mytopic (or update the topic property to match yours).
• Network access from Kestra to both the REST API and the Kafka brokers.

Secrets

This example connects to an open API and a broker address inline, so no secret() values are required. For production, move broker credentials and any API tokens into secrets and reference them, for example {{ secret('KAFKA_BOOTSTRAP_SERVERS') }}.

Quick start

1. Add this flow to a Kestra namespace.
2. Replace properties.bootstrap.servers with your cluster URL and confirm the topic exists.
3. Execute the flow manually.
4. Consume mytopic to confirm the API payload arrived as a JSON message.

How to extend

• Swap the api task URI for your own endpoint, or chain multiple requests.
• Send a list of maps to from to publish several messages in one run.
• Add a Schedule or webhook trigger to publish on a cadence or on demand.
• Transform the payload between fetch and produce, or route to different topics by key.

Links

• Plugin docs: https://kestra.io/plugins/plugin-kafka
• Orchestrate Kafka with Kestra: https://kestra.io/orchestration/kafka