Unity Catalog - Tables Trigger
CertifiedTrigger a flow when a Unity Catalog table appears in a schema
Unity Catalog - Tables Trigger
Trigger a flow when a Unity Catalog table appears in a schema
Periodically lists the tables of a schema and starts one execution for all the tables detected since the previous poll.
The set of known tables is persisted in the namespace KV Store, so the first evaluation reports every existing table as new when on is CREATE (the default).
Use on: UPDATE or on: CREATE_OR_UPDATE to also react to tables modified since the previous poll.
The detected tables are returned without their column definitions, use the table.Get task to read the full schema of a table.
type: io.kestra.plugin.databricks.unitycatalog.table.TriggerExamples
Start a flow when a new table appears in a schema.
id: databricks_uc_new_table
namespace: company.team
tasks:
- id: log_new_tables
type: io.kestra.plugin.core.log.Log
message: "New table(s) detected: {{ trigger.fullNames }}"
triggers:
- id: on_new_table
type: io.kestra.plugin.databricks.unitycatalog.table.Trigger
host: "{{ secret('DATABRICKS_HOST') }}"
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
catalogName: main
schemaName: landing_zone
interval: PT5M
Properties
catalogName *string
Catalog name
Name of the catalog that contains the monitored schema.
schemaName *string
Schema name
Name of the schema to monitor.
accountId string
Databricks account identifier
allowConcurrent boolean
falseSpecifies whether a trigger is allowed to start a new execution even if a previous run is still in progress.
authentication
Databricks authentication configuration
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
Authentication type
Azure client ID
Azure client secret
Azure tenant ID
Client ID
Client secret
Google credentials JSON
Google service account email
Password
Databricks personal access token
Username
configFile string
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one
host string
Databricks host
interval string
PT5MdurationInterval between polling.
The interval between 2 different polls of schedule, this can avoid to overload the remote system with too many calls. For most of the triggers that depend on external systems, a minimal interval must be at least PT30S. See ISO_8601 Durations for more information of available interval values.
on string
CREATECREATEUPDATECREATE_OR_UPDATETrigger event type
Defines when the trigger fires.
CREATE: only for newly discovered entities.UPDATE: only when an already-seen entity changes.CREATE_OR_UPDATE: fires on either event.
stateKey string
State key
JSON-type KV key for persisted state.
Default: <namespace>__<flowId>__<triggerId>
stateTtl string
State TTL
TTL for persisted state entries (e.g., PT24H, P7D).
stopAfter array
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDList of execution states after which a trigger should be stopped (a.k.a. disabled).
when string
trueA condition that determines whether the trigger should run.
A Pebble expression evaluated at trigger time. The trigger fires only when the expression evaluates to a truthy value (true, a non-empty string, a non-zero number). Use this to gate trigger execution on dynamic runtime values such as execution labels, flow variables, or environment conditions.
Outputs
fullNames array
Fully qualified names (catalog.schema.table) of the detected tables
size integer
Number of detected tables
tables array
Tables detected since the previous poll, as returned by Databricks, without their column definitions
