Vertica Batch

Vertica Batch
Certified
Bulk insert rows into Vertica using prepared statements

Reads ION-formatted data from Kestra internal storage and performs high-performance batch inserts using JDBC batch operations. Data is processed in chunks (default 1,000 rows) to optimize memory and performance. Optimized for Vertica's columnar storage. Supports auto-commit for databases without transaction support.

yaml
type: io.kestra.plugin.jdbc.vertica.Batch

Examples

Fetch rows from a table and bulk insert to another one.

yaml
id: vertica_batch_query
namespace: company.team

tasks:
  - id: query
    type: io.kestra.plugin.jdbc.vertica.Query
    url: jdbc:vertica://dev:56982/db
    username: "{{ secret('VERTICA_USERNAME') }}"
    password: "{{ secret('VERTICA_PASSWORD') }}"
    sql: |
      SELECT *
      FROM xref
      LIMIT 1500;
    fetchType: STORE

  - id: update
    type: io.kestra.plugin.jdbc.vertica.Batch
    from: "{{ outputs.query.uri }}"
    url: jdbc:vertica://prod:56982/db
    username: "{{ secret('VERTICA_USERNAME') }}"
    password: "{{ secret('VERTICA_PASSWORD') }}"
    sql: insert into xref values( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )

Fetch rows from a table and bulk insert to another one, without writing the INSERT statement.

yaml
    id: vertica_batch_query
    namespace: company.team

    tasks:
      - id: query
        type: io.kestra.plugin.jdbc.vertica.Query
        url: jdbc:vertica://dev:56982/db
        username: vertica_user
        password: "{{ secret('VERTICA_PASSWD') }}"
        sql: |
          SELECT *
          FROM xref
          LIMIT 1500;
        fetchType: STORE

      - id: update
        type: io.kestra.plugin.jdbc.vertica.Batch
        from: "{{ outputs.query.uri }}"
        url: jdbc:vertica://prod:56982/db
        username: vertica_user
        password: "{{ secret('VERTICA_PASSWD') }}"
        table: xref

Properties

from *string

Input file from internal storage

URI of the source file (kestra://) containing rows to insert

Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.

url *string

The JDBC URL to connect to the database

chunk integerstring

Default1000

Batch size per executeBatch call

Number of rows sent per JDBC batch before commit; default 1,000

columns array

SubTypestring

Columns bound to placeholders

Ordered column names matching ? placeholders; if omitted, placeholder count must match all columns in the input row

connectionPoolSize integerstring

Default10

Maximum number of pooled connections

Maximum connections held in the pool for a given URL and credentials. Default 10. Increase for flows that run many concurrent queries against the same database to avoid waiting for an available connection. Ignored when connectionPooling is false or for embedded drivers.

connectionPooling booleanstring

Defaulttrue

Reuse database connections via a connection pool

When true (default), connections are pooled and reused across executions, keyed by URL and credentials, removing the connect and TLS-handshake cost on each run. Set to false if your SQL relies on session state persisting on the connection (for example SET search_path, session-scoped temp tables or variables), since pooled connections are reused. Embedded drivers (DuckDB, SQLite, MS Access) never pool regardless of this setting.

inputHandling string

DefaultAUTO

Possible Values

AUTOSTREAMLOCAL

Input handling strategy

Controls how input is read during processing and retries. AUTO buffers small files locally (<= localBufferMaxBytes) and streams large files. STREAM always streams from internal storage. LOCAL always buffers input to a local temporary file before processing.

localBufferMaxBytes integerstring

Default104857600

Maximum number of bytes buffered locally

Used by AUTO and LOCAL input handling. In AUTO, files larger than this threshold are streamed. In LOCAL, files larger than this threshold fail fast.

maxRetries integerstring

Default3

Maximum number of retries for transient failures

Retries are attempted only for transient failures such as temporary I/O and recoverable SQL errors.

password string

The database user's password

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

resumeOnRetry booleanstring

Defaulttrue

Resume from the last successfully committed chunk on retry

retryBackoff string

DefaultPT1S

Delay between retry attempts

Uses ISO-8601 duration format, for example PT1S.

retryScope string

DefaultINPUT

Possible Values

NONEINPUTALL

Controls which failures are retried

INPUT retries input handling failures, ALL retries all retryable failures.

sql string

Parameterized INSERT statement to execute

Prepared INSERT with ? placeholders for each bound column. Example: INSERT INTO VALUES (?, ?, ?) for three columns; use column list if inserting a subset

table string

Table used to auto-discover columns

Retrieves column names from the given table when columns is empty. If sql is also omitted, an INSERT statement is generated automatically using the discovered columns

timeZoneId string

The time zone id to use for date/time manipulation. Default value is the worker's default time zone id

username string

The database user

Outputs

rowCount integer

Total rows read

updatedCount integer

Rows inserted or updated

Metrics

query counter

Unitqueries

The number of batch queries executed.

records counter

Unitrecords

The number of records processed.

updated counter

Unitrecords

The number of records updated.

Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

Vertica BatchCertifiedBulk insert rows into Vertica using prepared statements

More Plugins in this Category

Clickhouse

Weaviate

Vertica Batch
Certified
Bulk insert rows into Vertica using prepared statements