Deduplicate a file by retaining only the latest item for each extracted key.
The Deduplicate task involves reading the input file twice, rather than loading the entire file into memory.
The first iteration is used to build a deduplication map in memory containing the last lines observed for each key.
The second iteration is used to rewrite the file without the duplicates. The task must be used with this in mind.
type: "io.kestra.plugin.core.storage.DeduplicateItems"id: "deduplicateitems"
type: "io.kestra.plugin.core.storage.DeduplicateItems"
tasks:
- id: deduplicate
type: io.kestra.plugin.core.storage.DeduplicateItems
from: "{{ inputs.uri }}"
expr: "{{ key }}"
The 'pebble' expression to be used for extracting the deduplication key from each item.
The 'pebble' expression can be used for constructing a composite key.
The file to be deduplicated.
Must be a kestra:// internal storage URI.
The total number of items that was dropped by the task.
The number of distinct keys observed by the task.
The total number of items that was processed by the task.
The deduplicated file URI.