Apache Hudi Trigger
Certified
Wait for new commits in an Apache Hudi table
Apache Hudi Trigger
Wait for new commits in an Apache Hudi table
Polls the active timeline of an Apache Hudi table and triggers execution when new commits appear. The table must be at a remote path, a trigger has no shared working directory.
type: io.kestra.plugin.hudi.TriggerExamples
Poll for new Apache Hudi commits
id: hudi_trigger_flow
namespace: company.team
tasks:
- id: log_commit
type: io.kestra.plugin.core.log.Log
message: "New commit detected: {{ trigger.lastCommitTime }}"
triggers:
- id: watch_orders
type: io.kestra.plugin.hudi.Trigger
basePath: hdfs://namenode:8020/data/hudi/orders
tableName: orders
interval: PT1M
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-hudi --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
Properties
basePath *string
The base path of the Hudi table
Remote path of the Hudi table (e.g., hdfs://namenode: 8020/data/hudi/orders). Allowed remote schemes: hdfs, s3a, gs, abfs, abfss, wasb, wasbs. Local or file:// paths are not supported because a trigger has no shared working directory: it is new and empty on every check.
tableName *string
The name of the Hudi table
Target Apache Hudi table name.
allowConcurrent boolean
falseSpecifies whether a trigger is allowed to start a new execution even if a previous run is still in progress.
configuration object
Configuration properties for the Hudi connection
Key-value configuration map (e.g., Hadoop/AWS/GCS credentials, Hudi write configs). Allowed configuration keys start with hoodie., fs.s3a., fs.gs., fs.azure. or fs.abfs.. Within hoodie., only write and table tuning keys are accepted: record key, precombine and partition path fields, table name and type, index type, insert, upsert, bulk insert and delete parallelism, hoodie.cleaner.*, hoodie.clean.*, hoodie.compact.*, hoodie.clustering.*, hoodie.archive.*, hoodie.keep.*, hoodie.parquet.*, hoodie.logfile.*, hoodie.copyonwrite.* and hoodie.index.bloom.*. Other hoodie. keys are rejected, as is any key ending with .path, .dir or .pattern, or containing .class, .impl, .factory or classname. Within the fs.* prefixes, keys ending with .impl, .class, .classname or .factory, and keys containing provider, signers, interceptors, binding, keyfile or keytab, are not allowed; the credential and endpoint keys access.key, secret.key, session.token, endpoint, path.style.access and region (e.g. fs.s3a.access.key) are always accepted. fs.defaultFS and fs.viewfs.* are not allowed, the scheme comes from basePath. Values must be literal: variable expansion is not applied and values containing ${ are rejected.
interval string
PT5MdurationPolling interval
The duration between successive checks against the Hudi table timeline.
stopAfter array
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDList of execution states after which a trigger should be stopped (a.k.a. disabled).
when string
trueA condition that determines whether the trigger should run.
A Pebble expression evaluated at trigger time. The trigger fires only when the expression evaluates to a truthy value (true, a non-empty string, a non-zero number). Use this to gate trigger execution on dynamic runtime values such as execution labels, flow variables, or environment conditions.
Outputs
action string
Action type
The action type of the latest commit (e.g. commit, deltacommit).
basePath string
Base path
The base path of the triggered Hudi table.
lastCommitTime string
Latest commit instant timestamp
The timestamp of the latest completed commit on the active timeline.
tableName string
Table name
The name of the triggered Hudi table.