Apache Hudi Trigger

Apache Hudi Trigger

Certified

Wait for new commits in an Apache Hudi table

Polls the active timeline of an Apache Hudi table and triggers execution when new commits appear. The table must be at a remote path, a trigger has no shared working directory.

yaml
type: io.kestra.plugin.hudi.Trigger

Poll for new Apache Hudi commits

yaml
id: hudi_trigger_flow
namespace: company.team

tasks:
  - id: log_commit
    type: io.kestra.plugin.core.log.Log
    message: "New commit detected: {{ trigger.lastCommitTime }}"

triggers:
  - id: watch_orders
    type: io.kestra.plugin.hudi.Trigger
    basePath: hdfs://namenode:8020/data/hudi/orders
    tableName: orders
    interval: PT1M
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-hudi --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

The base path of the Hudi table

Remote path of the Hudi table (e.g., hdfs://namenode: 8020/data/hudi/orders). Allowed remote schemes: hdfs, s3a, gs, abfs, abfss, wasb, wasbs. Local or file:// paths are not supported because a trigger has no shared working directory: it is new and empty on every check.

The name of the Hudi table

Target Apache Hudi table name.

Defaultfalse

Specifies whether a trigger is allowed to start a new execution even if a previous run is still in progress.

Configuration properties for the Hudi connection

Key-value configuration map (e.g., Hadoop/AWS/GCS credentials, Hudi write configs). Allowed configuration keys start with hoodie., fs.s3a., fs.gs., fs.azure. or fs.abfs.. Within hoodie., only write and table tuning keys are accepted: record key, precombine and partition path fields, table name and type, index type, insert, upsert, bulk insert and delete parallelism, hoodie.cleaner.*, hoodie.clean.*, hoodie.compact.*, hoodie.clustering.*, hoodie.archive.*, hoodie.keep.*, hoodie.parquet.*, hoodie.logfile.*, hoodie.copyonwrite.* and hoodie.index.bloom.*. Other hoodie. keys are rejected, as is any key ending with .path, .dir or .pattern, or containing .class, .impl, .factory or classname. Within the fs.* prefixes, keys ending with .impl, .class, .classname or .factory, and keys containing provider, signers, interceptors, binding, keyfile or keytab, are not allowed; the credential and endpoint keys access.key, secret.key, session.token, endpoint, path.style.access and region (e.g. fs.s3a.access.key) are always accepted. fs.defaultFS and fs.viewfs.* are not allowed, the scheme comes from basePath. Values must be literal: variable expansion is not applied and values containing ${ are rejected.

DefaultPT5M
Formatduration

Polling interval

The duration between successive checks against the Hudi table timeline.

SubTypestring
Possible Values
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTED

List of execution states after which a trigger should be stopped (a.k.a. disabled).

Defaulttrue

A condition that determines whether the trigger should run.

A Pebble expression evaluated at trigger time. The trigger fires only when the expression evaluates to a truthy value (true, a non-empty string, a non-zero number). Use this to gate trigger execution on dynamic runtime values such as execution labels, flow variables, or environment conditions.

Action type

The action type of the latest commit (e.g. commit, deltacommit).

Base path

The base path of the triggered Hudi table.

Latest commit instant timestamp

The timestamp of the latest completed commit on the active timeline.

Table name

The name of the triggered Hudi table.