Clickhouse BulkInsert
CertifiedBulk insert rows into ClickHouse with JDBC batch
Clickhouse BulkInsert
Bulk insert rows into ClickHouse with JDBC batch
Inserts many rows into ClickHouse efficiently using a JDBC batch read from an internal storage file.
type: io.kestra.plugin.jdbc.clickhouse.BulkInsertExamples
Insert rows from another table to ClickHouse with async inserts
id: clickhouse_bulk_insert
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: bulk_insert
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
from: "{{ inputs.file }}"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
sql: INSERT INTO YourTable SETTINGS async_insert=1, wait_for_async_insert=1 VALUES( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
Insert into selected columns using async inserts
id: clickhouse_bulk_insert
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: bulk_insert
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
from: "{{ inputs.file }}"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
sql: INSERT INTO YourTable ( field1, field2, field3 ) SETTINGS async_insert=1, wait_for_async_insert=1 VALUES( ?, ?, ? )
Auto-generate INSERT using table column discovery
id: clickhouse_bulk_insert
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: bulk_insert
type: io.kestra.plugin.jdbc.clickhouse.BulkInsert
from: "{{ inputs.file }}"
url: jdbc:clickhouse://127.0.0.1:56982/
username: "{{ secret('CLICKHOUSE_USERNAME') }}"
password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
table: YourTable
Properties
from *string
Input file from internal storage
URI of the source file (kestra://) containing rows to insert
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
url *string
The JDBC URL to connect to the database
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150chunk integerstring
1000Batch size per executeBatch call
Number of rows sent per JDBC batch before commit; default 1,000
columns array
Columns bound to placeholders
Ordered column names matching ? placeholders; if omitted, placeholder count must match all columns in the input row
inputHandling string
AUTOAUTOSTREAMLOCALInput handling strategy
Controls how input is read during processing and retries.
AUTO buffers small files locally (<= localBufferMaxBytes) and streams large files.
STREAM always streams from internal storage.
LOCAL always buffers input to a local temporary file before processing.
localBufferMaxBytes integerstring
104857600Maximum number of bytes buffered locally
Used by AUTO and LOCAL input handling.
In AUTO, files larger than this threshold are streamed.
In LOCAL, files larger than this threshold fail fast.
maxRetries integerstring
3Maximum number of retries for transient failures
Retries are attempted only for transient failures such as temporary I/O and recoverable SQL errors.
password string
The database user's password
resumeOnRetry booleanstring
trueResume from the last successfully committed chunk on retry
retryBackoff string
PT1SDelay between retry attempts
Uses ISO-8601 duration format, for example PT1S.
retryScope string
INPUTNONEINPUTALLControls which failures are retried
INPUT retries input handling failures, ALL retries all retryable failures.
sql string
Parameterized INSERT statement to execute
Prepared INSERT with ? placeholders for each bound column. Example: INSERT INTO <table_name> VALUES (?, ?, ?) for three columns; use column list if inserting a subset
table string
Table used to auto-discover columns
Retrieves column names from the given table when columns is empty.
If sql is also omitted, an INSERT statement is generated automatically using the discovered columns
timeZoneId string
The time zone id to use for date/time manipulation. Default value is the worker's default time zone id
username string
The database user
Outputs
rowCount integer
Total rows read
updatedCount integer
Rows inserted or updated
Metrics
query counter
queriesThe number of batch queries executed.
records counter
recordsThe number of records processed.
updated counter
recordsThe number of records updated.
