Clickhouse BulkInsert

Clickhouse BulkInsert

Certified

Bulk insert rows into ClickHouse with JDBC batch

Inserts many rows into ClickHouse efficiently using a JDBC batch read from an internal storage file.

yaml
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert

Insert rows from another table to ClickHouse with async inserts

yaml
id: clickhouse_bulk_insert
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: bulk_insert
    type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
    from: "{{ inputs.file }}"
    url: jdbc:clickhouse://127.0.0.1:56982/
    username: "{{ secret('CLICKHOUSE_USERNAME') }}"
    password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
    sql: INSERT INTO YourTable SETTINGS async_insert=1, wait_for_async_insert=1 VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )

Insert into selected columns using async inserts

yaml
id: clickhouse_bulk_insert
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: bulk_insert
    type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
    from: "{{ inputs.file }}"
    url: jdbc:clickhouse://127.0.0.1:56982/
    username: "{{ secret('CLICKHOUSE_USERNAME') }}"
    password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
    sql: INSERT INTO YourTable ( field1, field2, field3 ) SETTINGS async_insert=1, wait_for_async_insert=1 VALUES( ?, ?, ? )

Auto-generate INSERT using table column discovery

yaml
id: clickhouse_bulk_insert
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: bulk_insert
    type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
    from: "{{ inputs.file }}"
    url: jdbc:clickhouse://127.0.0.1:56982/
    username: "{{ secret('CLICKHOUSE_USERNAME') }}"
    password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
    table: YourTable
Properties

Input file from internal storage

URI of the source file (kestra://) containing rows to insert

The JDBC URL to connect to the database

Default1000

Batch size per executeBatch call

Number of rows sent per JDBC batch before commit; default 1,000

SubTypestring

Columns bound to placeholders

Ordered column names matching ? placeholders; if omitted, placeholder count must match all columns in the input row

Default10

Maximum number of pooled connections

Maximum connections held in the pool for a given URL and credentials. Default 10. Increase for flows that run many concurrent queries against the same database to avoid waiting for an available connection. Ignored when connectionPooling is false or for embedded drivers.

Defaulttrue

Reuse database connections via a connection pool

When true (default), connections are pooled and reused across executions, keyed by URL and credentials, removing the connect and TLS-handshake cost on each run. Set to false if your SQL relies on session state persisting on the connection (for example SET search_path, session-scoped temp tables or variables), since pooled connections are reused. Embedded drivers (DuckDB, SQLite, MS Access) never pool regardless of this setting.

DefaultAUTO
Possible Values
AUTOSTREAMLOCAL

Input handling strategy

Controls how input is read during processing and retries. AUTO buffers small files locally (<= localBufferMaxBytes) and streams large files. STREAM always streams from internal storage. LOCAL always buffers input to a local temporary file before processing.

Default104857600

Maximum number of bytes buffered locally

Used by AUTO and LOCAL input handling. In AUTO, files larger than this threshold are streamed. In LOCAL, files larger than this threshold fail fast.

Default3

Maximum number of retries for transient failures

Retries are attempted only for transient failures such as temporary I/O and recoverable SQL errors.

The database user's password

Reference (ref) of the pluginDefaults to apply to this task.

Defaulttrue

Resume from the last successfully committed chunk on retry

DefaultPT1S

Delay between retry attempts

Uses ISO-8601 duration format, for example PT1S.

DefaultINPUT
Possible Values
NONEINPUTALL

Controls which failures are retried

INPUT retries input handling failures, ALL retries all retryable failures.

Parameterized INSERT statement to execute

Prepared INSERT with ? placeholders for each bound column. Example: INSERT INTO VALUES (?, ?, ?) for three columns; use column list if inserting a subset

Table used to auto-discover columns

Retrieves column names from the given table when columns is empty. If sql is also omitted, an INSERT statement is generated automatically using the discovered columns

The time zone id to use for date/time manipulation. Default value is the worker's default time zone id

The database user

Total rows read

Rows inserted or updated

Unitqueries

The number of batch queries executed.

Unitrecords

The number of records processed.

Unitrecords

The number of records updated.

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints