Databricks Zerobus WriteRecords

Databricks Zerobus WriteRecords

Certified

Push data directly into a Unity Catalog Delta table via Zerobus Ingest

This task sends data to a Databricks Delta table via the push-based Zerobus Ingest REST API.

It supports at-least-once delivery. If a task is retried after a partial failure, records that were already successfully accepted in previous chunks will be sent again and may be duplicated in the target table. Kestra's HTTP client may retry a failed request once on a server error, so with at-least-once delivery a chunk can be written twice.

The task authenticates using either OAuth client credentials or a Personal Access Token (PAT). An OAuth token is refreshed shortly before it expires, and a 401 on the ingest call triggers one refresh and one retry of that chunk. A PAT is never refreshed or retried. Note: Databricks documents short-lived OAuth tokens from a service principal as the only authentication for Zerobus Ingest. A PAT is sent as a Bearer token but has not been verified against Zerobus; prefer OAuth client credentials. The host property is required when using OAuth authentication. The default endpoint domain follows host: azuredatabricks.net for Azure, gcp.databricks.com for GCP, cloud.databricks.com otherwise.

Records are sent in chunks, bounded by conservative assumptions since the API docs do not specify a per-request limit.

Note: The inherited properties configFile and accountId are ignored by this task, and AbstractTask environment variable fallbacks are not honoured.

yaml
type: io.kestra.plugin.databricks.zerobus.WriteRecords

Push inline records to a Databricks Delta table

yaml
id: write_inline_records_to_databricks
namespace: company.team

tasks:
  - id: write_records
    type: io.kestra.plugin.databricks.zerobus.WriteRecords
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      clientId: "{{ secret('DATABRICKS_CLIENT_ID') }}"
      clientSecret: "{{ secret('DATABRICKS_CLIENT_SECRET') }}"
    workspaceId: "{{ secret('DATABRICKS_WORKSPACE_ID') }}"
    region: us-east-1
    catalog: main
    schema: events
    table: user_events
    records:
      - userId: usr_001
        event: page_view
      - userId: usr_002
        event: purchase

Push records from an internal storage file (Ion or JSON-lines)

yaml
id: write_file_to_databricks
namespace: company.team

inputs:
  - id: events_file
    type: FILE

tasks:
  - id: write_records
    type: io.kestra.plugin.databricks.zerobus.WriteRecords
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      clientId: "{{ secret('DATABRICKS_CLIENT_ID') }}"
      clientSecret: "{{ secret('DATABRICKS_CLIENT_SECRET') }}"
    workspaceId: "{{ secret('DATABRICKS_WORKSPACE_ID') }}"
    region: us-east-1
    catalog: main
    schema: events
    table: raw_events
    from: "{{ inputs.events_file }}"
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command: 

kestractl plugins get io.kestra.plugin:plugin-databricks --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version.

Properties

Unity Catalog catalog name

Schema (database) name

Target Delta table name

Databricks account identifier

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions
authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

Zerobus endpoint URL override

Replaces the computed Zerobus URL (https://<workspaceId>.zerobus.<region>.<cloud domain>). Use it for private connectivity, a proxy or tests.

Internal storage URI

URI of an Ion or JSON-lines file in Kestra internal storage. Each record must be one object; a single JSON array is not supported. Mutually exclusive with 'records'.

Databricks host

SubTypeobject

Inline records

List of JSON objects. Mutually exclusive with 'from'.

Cloud region

Cloud region used to build the default Zerobus endpoint, for example us-west-2 on AWS, eastus on Azure or us-east1 on GCP. Ignored when 'endpoint' is set. Required when 'endpoint' is not set.

Numeric workspace ID

Used to build the Zerobus endpoint and OAuth resource parameter.

Default0

Records count

Total number of records successfully sent to Zerobus Ingest

Number of records pushed to Zerobus Ingest

More Plugins in this Category

Dataform
Dataform

Tasks that run Dataform data transformations through the Dataform CLI.

Data

1 tasks

6 blueprints

Huawei
Huawei

Huawei plugin for Kestra

CloudDataInfrastructure

45 tasks

6 blueprints