Apache Hudi Cluster
Certified
Cluster an Apache Hudi table
Apache Hudi Cluster
Cluster an Apache Hudi table
Trigger clustering table service on an Apache Hudi table.
type: io.kestra.plugin.hudi.ClusterExamples
Cluster an Apache Hudi table
id: hudi_cluster_flow
namespace: company.team
tasks:
- id: cluster_table
type: io.kestra.plugin.hudi.Cluster
basePath: hdfs://namenode:8020/data/hudi/orders
tableName: orders
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-hudi --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
Properties
basePath *string
The base path of the Hudi table
Path where the Hudi table is located (e.g., orders, hdfs://namenode: 8020/data/hudi/orders). Allowed remote schemes: hdfs, s3a, gs, abfs, abfss, wasb, wasbs. Local paths (relative, absolute or file://) must be inside the working directory, and relative paths are resolved against it. A local table only lives as long as the working directory of the task. Remote s3a, gs and abfs paths only work when the matching Hadoop connector (hadoop-aws, gcs-connector, hadoop-azure) is added to the worker classpath, it is not bundled.
tableName *string
The name of the Hudi table
Target Apache Hudi table name.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150configuration object
Configuration properties for the Hudi connection
Key-value configuration map (e.g., Hadoop/AWS/GCS credentials, Hudi write configs). Allowed configuration keys start with hoodie., fs.s3a., fs.gs., fs.azure. or fs.abfs.. Within hoodie., only write and table tuning keys are accepted: record key, precombine and partition path fields, table name and type, index type, insert, upsert, bulk insert and delete parallelism, hoodie.cleaner.*, hoodie.clean.*, hoodie.compact.*, hoodie.clustering.*, hoodie.archive.*, hoodie.keep.*, hoodie.parquet.*, hoodie.logfile.*, hoodie.copyonwrite.* and hoodie.index.bloom.*. Other hoodie. keys are rejected, as is any key ending with .path, .dir or .pattern, or containing .class, .impl, .factory or classname. Within the fs.* prefixes, keys ending with .impl, .class, .classname or .factory, and keys containing provider, signers, interceptors, binding, keyfile or keytab, are not allowed; the credential and endpoint keys access.key, secret.key, session.token, endpoint, path.style.access and region (e.g. fs.s3a.access.key) are always accepted. fs.defaultFS and fs.viewfs.* are not allowed, the scheme comes from basePath. Values must be literal: variable expansion is not applied and values containing ${ are rejected.
Outputs
instantTime string
Clustering instant timestamp
The timestamp of the scheduled and executed clustering instant.