Clickhouse BulkInsert
CertifiedBulk insert rows into ClickHouse with JDBC batch
Clickhouse BulkInsert
Bulk insert rows into ClickHouse with JDBC batch
Inserts many rows into ClickHouse efficiently using a JDBC batch read from an internal storage file.
type: io.kestra.plugin.jdbc.clickhouse.BulkInsertExamples
Insert rows from another table to ClickHouse with async inserts
id: clickhouse_bulk_insert
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: bulk_insert
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
from: "{{ inputs.file }}"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
sql: INSERT INTO YourTable SETTINGS async_insert=1, wait_for_async_insert=1 VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
Insert into selected columns using async inserts
id: clickhouse_bulk_insert
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: bulk_insert
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
from: "{{ inputs.file }}"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
sql: INSERT INTO YourTable ( field1, field2, field3 ) SETTINGS async_insert=1, wait_for_async_insert=1 VALUES( ?, ?, ? )
Auto-generate INSERT using table column discovery
id: clickhouse_bulk_insert
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: bulk_insert
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
from: "{{ inputs.file }}"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
table: YourTable
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-jdbc-clickhouse --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version.
Properties
from *string
Input file from internal storage
URI of the source file (kestra://) containing rows to insert
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
url *string
The JDBC URL to connect to the database
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150chunk integerstring
1000Batch size per executeBatch call
Number of rows sent per JDBC batch before commit; default 1,000
columns array
Columns bound to placeholders
Ordered column names matching ? placeholders; if omitted, placeholder count must match all columns in the input row
inputHandling string
AUTOAUTOSTREAMLOCALInput handling strategy
Controls how input is read during processing and retries.
AUTO buffers small files locally (<= localBufferMaxBytes) and streams large files.
STREAM always streams from internal storage.
LOCAL always buffers input to a local temporary file before processing.
localBufferMaxBytes integerstring
104857600Maximum number of bytes buffered locally
Used by AUTO and LOCAL input handling.
In AUTO, files larger than this threshold are streamed.
In LOCAL, files larger than this threshold fail fast.
maxRetries integerstring
3Maximum number of retries for transient failures
Retries are attempted only for transient failures such as temporary I/O and recoverable SQL errors.
outputStatement booleanstring
falseExpose the executed SQL statement in the outputs
When true, the INSERT statement is exposed in the statement output, truncated to 64 KiB.
Disabled by default so that SQL text never leaks into outputs unless requested
password string
The database user's password
resumeOnRetry booleanstring
trueResume from the last successfully committed chunk on retry
retryBackoff string
PT1SDelay between retry attempts
Uses ISO-8601 duration format, for example PT1S.
retryScope string
INPUTNONEINPUTALLControls which failures are retried
INPUT retries input handling failures, ALL retries all retryable failures.
sql string
Parameterized INSERT statement to execute
Prepared INSERT with ? placeholders for each bound column. Example: INSERT INTO <table_name> VALUES (?, ?, ?) for three columns; use column list if inserting a subset
table string
Table used to auto-discover columns
Retrieves column names from the given table when columns is empty.
If sql is also omitted, an INSERT statement is generated automatically using the discovered columns
timeZoneId string
The time zone id to use for date/time manipulation. Default value is the worker's default time zone id
username string
The database user
Outputs
durationMs integer
Batch execution duration in milliseconds
Includes input handling and retries
rowCount integer
Total rows read
rowsAffected integer
Number of rows affected by the batch
Sum of the update counts reported by the JDBC driver for every executed batch
statement string
Executed SQL statement
Only populated when outputStatement is true; truncated to 64 KiB
tables array
Tables written by the statement
Detected from the SQL text, so this is a best effort.
Limited to 50 entries, see tablesTruncated
io.kestra.plugin.jdbc.SqlTableExtractor-TableReference
READWRITECREATEDROPHow the statement uses the table
One of READ, WRITE, CREATE or DROP.
MEDIUMLOWHow reliable the detection is
MEDIUM when detected from the SQL text, LOW for temporary tables and ambiguous names.
Database name
Null when the statement does not qualify the table with a database.
TABLEFILEURLKind of source
Only set on DuckDB: TABLE, FILE or URL (files read through read_parquet, read_csv, ATTACH and quoted paths).
Table name
For DuckDB file reads, the file path or URL.
Schema name
Null when the statement does not qualify the table with a schema.
tablesTruncated boolean
Whether the tables list was cut at 50 entries
updatedCount integer
Rows inserted or updated
Kept for compatibility, prefer rowsAffected
warnings integer
Number of SQL warnings raised by the batch
Count of the SQLWarning chain reported by the driver across all executed batches
Metrics
duration timer
The duration of the SQL execution, including reading the result rows.
query counter
queriesThe number of batch queries executed.
records counter
recordsThe number of records processed.
rows.affected counter
rowsThe number of rows affected, as reported by the JDBC driver.
updated counter
recordsThe number of records updated.
