Blueprints

Use Pulsar Realtime Trigger to push events into MySQL

About this blueprint

Realtime SQL

This flow will:

  1. Get triggered every time the event lands in Apache Pulsar topic
  2. The flow will push the data into a table in MySQL database

To setup Apache Pulsar locally, you can install the standalone cluster or docker cluster for Apache Pulsar. You can run the following commands to create the topic, and produce data to the topic:

  1. Setup a tenant bin/pulsar-admin tenants create apache
  2. Create a namespace bin/pulsar-admin namespaces create apache/pulsar
  3. Create a topic bin/pulsar-admin topics create-partitioned-topic apache/pulsar/logs -p 4
  4. Produce data to topic bin/pulsar-client produce apache/pulsar/logs -m '--Hello World--' -n 1

To setup MySQL server locally, follow the official installation steps using docker.

yaml
id: pulsar_realtime_trigger
namespace: company.team

tasks:
  - id: create_mysql_table
    type: io.kestra.plugin.jdbc.mysql.Query
    sql: |
      create table if not exists logs (
        log_id INT AUTO_INCREMENT,
        message VARCHAR(500),
        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
        PRIMARY KEY(log_id)
      )
      
  - id: insert_into_logs_table
    type: io.kestra.plugin.jdbc.mysql.Query
    sql: insert into logs(message) values("{{ trigger.value }}")
    
triggers:
  - id: realtime_trigger
    type: io.kestra.plugin.pulsar.RealtimeTrigger
    topic: apache/pulsar/logs
    uri: pulsar://localhost:26650
    subscriptionName: kestra_trigger_sub

pluginDefaults:
  - type: "io.kestra.plugin.jdbc.mysql.Query" 
    values:
      url: jdbc:mysql://localhost:3306/kestra
      username: mysql_user
      password: mysql_passwd

Query

Realtime Trigger

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra