Grok TransformValue
CertifiedParse and structure raw JSON string using Grok expressions
Grok TransformValue
Parse and structure raw JSON string using Grok expressions
The TransformValue task is similar to the famous Logstash Grok filter from the ELK stack.
It is particularly useful for transforming unstructured data such as logs into a structured, indexable, and queryable data structure.
The TransformValue ships with all the default patterns as defined You can find them here: https://github.com/kestra-io/plugin-transform/tree/main/plugin-transform-grok/src/main/resources/patterns.
type: io.kestra.plugin.transform.grok.TransformValueExamples
Consume, parse, and structure logs events from Kafka topic.
id: grok_transform_value
namespace: company.team
tasks:
- id: transform_value
type: io.kestra.plugin.transform.grok.TransformValue
pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
from: "{{ trigger.value }}"
- id: log_on_warn
type: io.kestra.plugin.core.flow.If
condition: "{{ grok.value['LOGLEVEL'] == 'ERROR' }}"
then:
- id: when_true
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.transform_value.value }}"
triggers:
- id: realtime_trigger
type: io.kestra.plugin.kafka.RealtimeTrigger
topic: test_kestra
properties:
bootstrap.servers: localhost:9092
serdeProperties:
schema.registry.url: http://localhost:8085
keyDeserializer: STRING
valueDeserializer: STRING
groupId: kafkaConsumerGroupId
Properties
from *string
The value to parse
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150breakOnFirstMatch booleanstring
trueIf true, break on first match
The first successful match by grok will result in the task being finished. Set to false if you want the task to try all configured patterns.
keepEmptyCaptures booleanstring
falseIf true, keep empty captures
When an optional field cannot be captured, the empty field is retained in the output. Set false if you want empty optional fields to be filtered out.
namedCapturesOnly booleanstring
trueIf true, only store named captures from grok
pattern string
The Grok pattern to match
patternDefinitions object
Custom pattern definitions
A map of pattern-name and pattern pairs defining custom patterns to be used by the current tasks. Patterns matching existing names will override the pre-existing definition.
patterns array
The list of Grok patterns to match
patternsDir array
List of user-defined pattern directories
Directories must be paths relative to the working directory.
