TransformValue

type: "io.kestra.plugin.transform.grok.TransformValue"

Parse arbitrary text and structure it using Grok expressions.

The TransformValue task is similar to the famous Logstash Grok filter from the ELK stack. It is particularly useful for transforming unstructured data such as logs into a structured, indexable, and queryable data structure.

The TransformValue ships with all the default patterns as defined You can find them here: https://github.com/kestra-io/plugin-transform/tree/main/plugin-transform-grok/src/main/resources/patterns.

Examples

Consume, parse, and structure logs events from Kafka topic.

id: grok
namespace: myteam

tasks:
- id: grok
  type: io.kestra.plugin.transform.grok.TransformValue
  pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
  from: "{{ trigger.value }}"

- id: log_on_warn
  type: io.kestra.plugin.core.flow.If
  condition: "{{ grok.value['LOGLEVEL'] == 'ERROR' }}"
  then:
    - id: when_true
      type: io.kestra.plugin.core.log.Log
      message: "{{ outputs.grok.value }}"

triggers:
- id: realtime_trigger
  type: io.kestra.plugin.kafka.RealtimeTrigger
  topic: test_kestra
  properties:
    bootstrap.servers: localhost:9092
  serdeProperties:
    schema.registry.url: http://localhost:8085
    keyDeserializer: STRING
    valueDeserializer: STRING
  groupId: kafkaConsumerGroupId

Properties

from

• Type: string
• Dynamic: ✔️
• Required: ✔️

The value to parse.

breakOnFirstMatch

• Type: boolean
• Dynamic: ❌
• Required: ❌
• Default: true

If true, break on first match.

The first successful match by grok will result in the task being finished. Set to false if you want the task to try all configured patterns.

namedCapturesOnly

• Type: boolean
• Dynamic: ❌
• Required: ❌
• Default: true

If true, only store named captures from grok.

pattern

• Type: string
• Dynamic: ❌
• Required: ❌

The Grok pattern to match.

patternDefinitions

• Type: object
• SubType: string
• Dynamic: ❌
• Required: ❌

Custom pattern definitions.

A map of pattern-name and pattern pairs defining custom patterns to be used by the current tasks. Patterns matching existing names will override the pre-existing definition.

patterns

• Type: array
• SubType: string
• Dynamic: ❌
• Required: ❌

The list of Grok patterns to match.

patternsDir

• Type: array
• SubType: string
• Dynamic: ❌
• Required: ❌

List of user-defined pattern directories.

Directories must be paths relative to the working directory.

Outputs

Definitions