Apache Hudi Commits
Certified
List timeline commits for an Apache Hudi table
Apache Hudi Commits
List timeline commits for an Apache Hudi table
Read the active timeline of an Apache Hudi table and return commit metadata.
type: io.kestra.plugin.hudi.CommitsExamples
List timeline commits for an Apache Hudi table
id: hudi_commits_flow
namespace: company.team
tasks:
- id: list_commits
type: io.kestra.plugin.hudi.Commits
basePath: hdfs://namenode:8020/data/hudi/orders
tableName: orders
fetchType: FETCH
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-hudi --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
Properties
basePath *string
The base path of the Hudi table
Path where the Hudi table is located (e.g., orders, hdfs://namenode: 8020/data/hudi/orders). Allowed remote schemes: hdfs, s3a, gs, abfs, abfss, wasb, wasbs. Local paths (relative, absolute or file://) must be inside the working directory, and relative paths are resolved against it. A local table only lives as long as the working directory of the task. Remote s3a, gs and abfs paths only work when the matching Hadoop connector (hadoop-aws, gcs-connector, hadoop-azure) is added to the worker classpath, it is not bundled.
tableName *string
The name of the Hudi table
Target Apache Hudi table name.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150configuration object
Configuration properties for the Hudi connection
Key-value configuration map (e.g., Hadoop/AWS/GCS credentials, Hudi write configs). Allowed configuration keys start with hoodie., fs.s3a., fs.gs., fs.azure. or fs.abfs.. Within hoodie., only write and table tuning keys are accepted: record key, precombine and partition path fields, table name and type, index type, insert, upsert, bulk insert and delete parallelism, hoodie.cleaner.*, hoodie.clean.*, hoodie.compact.*, hoodie.clustering.*, hoodie.archive.*, hoodie.keep.*, hoodie.parquet.*, hoodie.logfile.*, hoodie.copyonwrite.* and hoodie.index.bloom.*. Other hoodie. keys are rejected, as is any key ending with .path, .dir or .pattern, or containing .class, .impl, .factory or classname. Within the fs.* prefixes, keys ending with .impl, .class, .classname or .factory, and keys containing provider, signers, interceptors, binding, keyfile or keytab, are not allowed; the credential and endpoint keys access.key, secret.key, session.token, endpoint, path.style.access and region (e.g. fs.s3a.access.key) are always accepted. fs.defaultFS and fs.viewfs.* are not allowed, the scheme comes from basePath. Values must be literal: variable expansion is not applied and values containing ${ are rejected.
fetchType string
FETCHSTOREFETCHFETCH_ONENONEThe fetch type
How to return the fetched commits: FETCH (default, returns list in memory), FETCH_ONE (single row), or STORE (stores as internal storage file).
Outputs
row object
Single commit
Single (latest) commit metadata object when fetchType is FETCH_ONE.
rows array
List of commits
List of commit metadata objects when fetchType is FETCH.
size integer
Number of fetched commits
Total number of commits retrieved from the timeline.
uri string
Internal storage URI
URI of stored commits file when fetchType is STORE.