Apache Hudi Delete

Apache Hudi Delete

Certified

Delete data from an Apache Hudi table

Delete records from an Apache Hudi table using HoodieJavaWriteClient. On a partitioned table, from must contain the partition path of each record, otherwise the task fails instead of deleting nothing.

yaml
type: io.kestra.plugin.hudi.Delete

Delete records from an Apache Hudi table

yaml
id: hudi_delete_flow
namespace: company.team

tasks:
  - id: delete_records
    type: io.kestra.plugin.hudi.Delete
    basePath: hdfs://namenode:8020/data/hudi/orders
    tableName: orders
    recordKeyField: order_id
    from:
      - order_id: 101
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-hudi --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

The base path of the Hudi table

Path where the Hudi table is located (e.g., orders, hdfs://namenode: 8020/data/hudi/orders). Allowed remote schemes: hdfs, s3a, gs, abfs, abfss, wasb, wasbs. Local paths (relative, absolute or file://) must be inside the working directory, and relative paths are resolved against it. A local table only lives as long as the working directory of the task. Remote s3a, gs and abfs paths only work when the matching Hadoop connector (hadoop-aws, gcs-connector, hadoop-azure) is added to the worker classpath, it is not bundled.

Source keys or records

Keys or records to delete. Can be a list of maps, strings, or a Kestra internal storage URI (kestra://...). When partitionPathField is set (partitioned table) every entry must be a record containing both the record key field and the partition path field, because a bare key cannot be located; the task fails otherwise. Bare keys only work on non-partitioned tables.

Record key field

Field name to extract the record key from incoming records.

The name of the Hudi table

Target Apache Hudi table name.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Configuration properties for the Hudi connection

Key-value configuration map (e.g., Hadoop/AWS/GCS credentials, Hudi write configs). Allowed configuration keys start with hoodie., fs.s3a., fs.gs., fs.azure. or fs.abfs.. Within hoodie., only write and table tuning keys are accepted: record key, precombine and partition path fields, table name and type, index type, insert, upsert, bulk insert and delete parallelism, hoodie.cleaner.*, hoodie.clean.*, hoodie.compact.*, hoodie.clustering.*, hoodie.archive.*, hoodie.keep.*, hoodie.parquet.*, hoodie.logfile.*, hoodie.copyonwrite.* and hoodie.index.bloom.*. Other hoodie. keys are rejected, as is any key ending with .path, .dir or .pattern, or containing .class, .impl, .factory or classname. Within the fs.* prefixes, keys ending with .impl, .class, .classname or .factory, and keys containing provider, signers, interceptors, binding, keyfile or keytab, are not allowed; the credential and endpoint keys access.key, secret.key, session.token, endpoint, path.style.access and region (e.g. fs.s3a.access.key) are always accepted. fs.defaultFS and fs.viewfs.* are not allowed, the scheme comes from basePath. Values must be literal: variable expansion is not applied and values containing ${ are rejected.

Partition path field

Field name to extract the partition path. Defaults to empty for non-partitioned tables.

SubTypestring

Affected partition paths

List of partition paths modified during the delete operation.

Commit instant timestamp

The instant timestamp generated for this commit on the Hudi active timeline.

Number of records deleted

Number of records actually deleted, as reported by the Hudi write statuses. Keys that do not exist are not counted.