Apache Hudi
Certified
Apache Hudi plugin for Kestra
Apache Hudi (Hadoop Upsert Delete and Incremental) plugin for Kestra providing stream and batch table services, upserts, deletes, clustering, compaction, cleaning, and active timeline polling triggers.
Apache Hudi
Apache Hudi plugin for Kestra
Apache Hudi (Hadoop Upsert Delete and Incremental) plugin for Kestra providing stream and batch table services, upserts, deletes, clustering, compaction, cleaning, and active timeline polling triggers.
tasks
How to use the Apache Hudi plugin
This plugin reads, writes and maintains Apache Hudi tables from Kestra with the Hudi Java client, without a Spark or Flink cluster.
Authentication
Every task and the trigger take the same connection properties:
basePath(required): table location, for examplehdfs://namenode: 8020/data/hudi/orders. Allowed remote schemes:hdfs,s3a,gs,abfs,abfss,wasb,wasbs. A local path (relative, absolute orfile://) must be inside the working directory and is resolved against it.tableName(required): name of the Hudi table.configuration(optional, secret): map of Hadoop and Hudi settings, for example storage credentials orhoodie.*write configs. Allowed configuration keys start withhoodie.,fs.s3a.,fs.gs.,fs.azure.orfs.abfs.. Withinhoodie., only write and table tuning keys are accepted: record key, precombine and partition path fields, table name and type, index type, insert, upsert, bulk insert and delete parallelism, and thehoodie.cleaner.*,hoodie.clean.*,hoodie.compact.*,hoodie.clustering.*,hoodie.archive.*,hoodie.keep.*,hoodie.parquet.*,hoodie.logfile.*,hoodie.copyonwrite.*andhoodie.index.bloom.*groups. Otherhoodie.keys are rejected, as is any key ending with.path,.diror.pattern, or containing.class,.impl,.factoryorclassname. Within thefs.*prefixes, keys ending with.impl,.class,.classnameor.factory, and keys containingprovider,signers,interceptors,binding,keyfileorkeytab, are not allowed, except the credential and endpoint keysaccess.key,secret.key,session.token,endpoint,path.style.accessandregion.fs.defaultFSandfs.viewfs.*are not allowed, the scheme comes frombasePath. Values must be literal: variable expansion is not applied and values containing${are rejected.
configuration is marked as a secret and is hidden from logs. Store credentials in secrets and reference them with {{ secret('NAME') }}. To share the same connection across flows, use plugin defaults.
Only the file systems present on the plugin classpath can be used. HDFS paths and local paths work out of the box; s3a://, gs:// and abfs:// paths only work when the matching Hadoop connector (hadoop-aws, gcs-connector, hadoop-azure) is added to the worker classpath, it is not bundled.
A local table lives and dies with the working directory of the task. To use one table from several tasks, wrap them in io.kestra.plugin.core.flow.WorkingDirectory so they share the directory.
Tasks
Write
Upsert and Insert share these properties:
from(required): inline list of records, or akestra://URI of a file with one JSON record per line.recordKeyField(required): field used as the Hudi record key. A record without this field fails the task.partitionPathField: field used as the partition path, empty for a non-partitioned table.schema: Avro schema as a JSON string. When omitted, it is inferred from the records.precombineField: required forUpsert, optional forInsert.
Both tasks create the table (copy on write) when basePath has none, and fail when Hudi reports a write error. Outputs: recordCount, commitInstantTime, affectedPartitions.
Delete removes records by key. from accepts a list of keys, a list of records or a kestra:// URI, and recordKeyField is required. Its recordCount only counts records that were actually deleted.
Read
Commits lists the completed commits of the active timeline with instantTime, action, state and totalRecords. totalRecords is the number of records written by that commit, including records rewritten by copy on write file rewrites, not the size of the table after the commit (for example a delete commit can show 0). Use fetchType (FETCH, FETCH_ONE or STORE) to choose how rows are returned.
Maintenance
Compact, Cluster and Clean run the matching Hudi table service. They never create a table: when basePath has no Hudi table, the task fails.
Compactonly does work on merge on read tables created outside this plugin, since this plugin creates copy on write tables only. It returns a nullinstantTimewhen nothing is scheduled.Clusterreturns a nullinstantTimewhen no clustering plan is scheduled.CleanreturnsinstantTimeandfilesCleaned.
Killing a Compact or Cluster execution does not roll back the instant in progress. A crashed or killed Compact can leave an inflight compaction instant on the timeline. Because it may be running in another engine, Compact never rolls it back or executes it, and fails on later runs until the instant is completed or rolled back. To clear it, make sure nothing is still running it, then roll it back with Hudi tooling, for example compaction unschedule or commit rollback in hudi-cli. A plan left in the requested state is executed by the next Compact run.
Delete
Delete locates a record by its key and partition path. For a partitioned table, set partitionPathField and make from contain records with both the record key field and the partition path field; bare keys fail the task with an error naming from and partitionPathField. Bare keys work on non-partitioned tables only, so leave partitionPathField unset for those.
Triggers
Trigger needs a remote basePath (one of the allowed remote schemes), because a trigger has no shared working directory: it is new and empty on every check, so a local table could never be found. Trigger polls the active timeline every interval (default PT5M) and starts an execution when a commit newer than the last seen one appears. The last seen commit is stored in the namespace KV store per flow, trigger and table. A missing table is skipped; any other error, such as bad credentials, fails the poll.
Outputs: lastCommitTime, action, tableName, basePath.
id: hudi_new_commit
namespace: company.analytics
tasks:
- id: notify
type: io.kestra.plugin.core.log.Log
message: "New Hudi commit on orders: {{ trigger.lastCommitTime }}"
triggers:
- id: wait_for_commit
type: io.kestra.plugin.hudi.Trigger
basePath: hdfs://namenode: 8020/data/hudi/orders
tableName: orders
interval: PT2M