Grok TransformItems
CertifiedParse and structure items from a UTF-8 text or Ion file using Grok expressions
Grok TransformItems
Parse and structure items from a UTF-8 text or Ion file using Grok expressions
The TransformItems task is similar to the famous Logstash Grok filter from the ELK stack.
It is particularly useful for transforming unstructured data such as logs into a structured, indexable, and queryable data structure.
The source file is read line by line and each line is evaluated independently. Plain-text files are supported alongside Ion/JSON content, while non-matching rows are ignored. Multi-line records such as stack traces are therefore treated as separate entries rather than being preserved as a single item.
The TransformItems ships with all the default patterns as defined. You can find them here: https://github.com/kestra-io/plugin-transform/tree/main/plugin-transform-grok/src/main/resources/patterns.
type: io.kestra.plugin.transform.grok.TransformItemsExamples
Read a plain-text log file from internal storage and parse each line with Grok.
id: grok_transform_items
namespace: company.team
tasks:
- id: read_log
type: io.kestra.plugin.core.storage.Read
uri: "kestra://company.team/production/app.log"
- id: transform_items
type: io.kestra.plugin.transform.grok.TransformItems
pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
from: "{{ outputs.read_log.uri }}"
Properties
from *Requiredstring
The file to be transformed
Must be a kestra:// internal storage URI pointing to a UTF-8 plain-text or Ion/JSON file. Each line is matched independently, so plain-text logs are processed one physical line at a time.
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
breakOnFirstMatch booleanstring
trueIf true, break on first match
The first successful match by grok will result in the task being finished. Set to false if you want the task to try all configured patterns.
keepEmptyCaptures booleanstring
falseIf true, keep empty captures
When an optional field cannot be captured, the empty field is retained in the output. Set false if you want empty optional fields to be filtered out.
namedCapturesOnly booleanstring
trueIf true, only store named captures from grok
pattern string
The Grok pattern to match
patternDefinitions object
Custom pattern definitions
A map of pattern-name and pattern pairs defining custom patterns to be used by the current tasks. Patterns matching existing names will override the pre-existing definition.
patterns array
The list of Grok patterns to match
patternsDir array
List of user-defined pattern directories
Directories must be paths relative to the working directory.
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
