Use Pulsar Realtime Trigger to push events into MySQL

Stream events from an Apache Pulsar topic into a MySQL table the moment they arrive. This event-driven blueprint subscribes to a Pulsar topic and writes every consumed message into a relational table, giving you a low-latency ingestion path between your messaging layer and your operational database. It solves the common problem of bridging a high-throughput streaming system and a SQL store without standing up a separate stream-processing service or polling job.

How it works

1. The realtime_trigger trigger of type io.kestra.plugin.pulsar.RealtimeTrigger subscribes to the Pulsar topic apache/pulsar/logs over pulsar://localhost:26650 using the subscription name kestra_trigger_sub. It fires one flow execution per message as events arrive.
2. The create_mysql_table task (io.kestra.plugin.jdbc.mysql.Query) runs an idempotent create table if not exists logs (...) statement so the target table is always present.
3. The insert_into_logs_table task (io.kestra.plugin.jdbc.mysql.Query) inserts the message payload, read from {{ trigger.value }}, into the logs table.

Database connection details (url, username, password) are supplied once through pluginDefaults for the MySQL Query type.

What you get

• Real-time, per-message ingestion from Pulsar into MySQL.
• An auto-created target table so the flow is safe to run from a clean database.
• A clear, auditable execution per event with full input and output capture.
• A template you can point at any Pulsar topic and any JDBC target.

Who it's for

• Data engineers building streaming-to-SQL ingestion pipelines.
• Platform teams that need event-driven loading without custom consumer code.
• Developers prototyping Pulsar-backed event flows against a relational store.

Why orchestrate this with Kestra

Pulsar moves messages, but it does not orchestrate what happens to them. Kestra turns each message into a governed execution: the RealtimeTrigger reacts to events instead of polling, failed inserts can be retried declaratively, and every run is recorded with inputs, outputs, and logs for full lineage. The entire pipeline lives in version-controlled YAML, so the consumer logic, the table schema, and the connection defaults are reviewed and deployed like code rather than hidden inside a standalone consumer application.

Prerequisites

• A running Apache Pulsar cluster reachable at the configured uri.
• A MySQL database reachable at the configured JDBC url.
• The Pulsar and JDBC MySQL plugins (bundled with Kestra).

Secrets

This blueprint does not reference any {{ secret('NAME') }} values. The MySQL credentials are set inline in pluginDefaults. For production, replace the inline username and password with secrets, for example {{ secret('MYSQL_USERNAME') }} and {{ secret('MYSQL_PASSWORD') }}.

Quick start

1. Stand up Apache Pulsar locally as a standalone cluster or via Docker.
2. Create the tenant, namespace, and topic: bin/pulsar-admin tenants create apache, bin/pulsar-admin namespaces create apache/pulsar, then bin/pulsar-admin topics create-partitioned-topic apache/pulsar/logs -p 4.
3. Start a MySQL server (for example the official Docker image) and adjust the pluginDefaults connection values.
4. Add the blueprint to Kestra and enable the flow.
5. Produce a message: bin/pulsar-client produce apache/pulsar/logs -m '--Hello World--' -n 1, then watch a new row appear in the logs table.

How to extend

• Point topic, uri, and subscriptionName at your own Pulsar topic.
• Parse JSON payloads and map fields to dedicated columns instead of a single message field.
• Swap the MySQL tasks for Postgres, SQL Server, or another JDBC target.
• Add validation, enrichment, or alerting tasks between the trigger and the insert.
• Configure retries on the insert task to handle transient database errors.

Links

• Apache Pulsar plugin documentation