Apache Flink SubmitSql

Apache Flink SubmitSql

Certified

Execute SQL via Flink SQL Gateway

Runs a SQL statement through the Flink SQL Gateway without uploading a JAR. Respects session settings and waits until the operation reaches a terminal state.

yaml
type: io.kestra.plugin.flink.SubmitSql

Execute a streaming SQL query

yaml
id: flink-sql-streaming
namespace: company.team

tasks:
  - id: run-sql
    type: io.kestra.plugin.flink.SubmitSql
    gatewayUrl: "http://flink-sql-gateway:8083"
    statement: |
      INSERT INTO enriched_orders
      SELECT o.order_id, o.customer_id, c.name, o.amount, o.order_time
      FROM orders o
      JOIN customers c ON o.customer_id = c.id
    sessionConfig:
      catalog: "default_catalog"
      database: "default_database"
      configuration:
        execution.runtime-mode: "streaming"
        execution.checkpointing.interval: "30s"

# The SQL Gateway must be reachable; the configured catalog and source/target tables must exist.

Execute a batch SQL query

yaml
id: flink-sql-batch
namespace: company.team

tasks:
  - id: run-batch-sql
    type: io.kestra.plugin.flink.SubmitSql
    gatewayUrl: "http://flink-sql-gateway:8083"
    statement: |
      CREATE TABLE daily_summary AS
      SELECT DATE(order_time) as order_date,
             COUNT(*) as order_count,
             SUM(amount) as total_amount
      FROM orders
      WHERE order_time >= '2024-01-01'
      GROUP BY DATE(order_time)
    sessionConfig:
      configuration:
        execution.runtime-mode: "batch"

# The SQL Gateway must be reachable; the referenced source table and required SQL objects must exist.
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-flink --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version.

Properties

SQL Gateway URL

Base URL of the Flink SQL Gateway (e.g., http://flink-sql-gateway: 8083).

SQL statement

SQL statement to execute; supports DDL and DML.

SubTypestring

Acceptable terminal states

Operation states treated as success. For streaming include RUNNING to keep the session open. Defaults to ['FINISHED', 'RUNNING'].

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Default30

Connection timeout

Connection timeout in seconds, minimum 1; defaults to 30.

Session configuration

Session options such as catalog, database, and configuration properties applied before execution.

Definitions
catalogstring

Catalog

Catalog to use for the SQL session.

configurationobject
SubTypestring

Configuration

Flink configuration options applied to the SQL session.

databasestring

Database

Database to use for the SQL session.

Session name

Optional session name; a temporary session is created when omitted.

Default300

Statement timeout

Execution timeout in seconds for the SQL statement; defaults to 300.

Operation handle

Unique identifier for the executed SQL operation.

Result count

Number of rows affected or returned; -1 when the gateway does not provide it.

Session handle

Identifier for the SQL Gateway session used.

Operation status

Final status reported by the SQL Gateway.