Grok TransformValue

Grok TransformValue

Certified

Parse and structure raw JSON string using Grok expressions

The TransformValue task is similar to the famous Logstash Grok filter from the ELK stack. It is particularly useful for transforming unstructured data such as logs into a structured, indexable, and queryable data structure.

The TransformValue ships with all the default patterns as defined You can find them here: https://github.com/kestra-io/plugin-transform/tree/main/plugin-transform-grok/src/main/resources/patterns.

yaml
type: io.kestra.plugin.transform.grok.TransformValue

Consume, parse, and structure logs events from Kafka topic.

yaml
id: grok_transform_value
namespace: company.team

tasks:
  - id: transform_value
    type: io.kestra.plugin.transform.grok.TransformValue
    pattern: "%{TIMESTAMP_ISO8601:logdate} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
    from: "{{ trigger.value }}"

  - id: log_on_warn
    type: io.kestra.plugin.core.flow.If
    condition: "{{ grok.value['LOGLEVEL'] == 'ERROR' }}"
    then:
      - id: when_true
        type: io.kestra.plugin.core.log.Log
        message: "{{ outputs.transform_value.value }}"

triggers:
  - id: realtime_trigger
    type: io.kestra.plugin.kafka.RealtimeTrigger
    topic: test_kestra
    properties:
      bootstrap.servers: localhost:9092
    serdeProperties:
      schema.registry.url: http://localhost:8085
      keyDeserializer: STRING
      valueDeserializer: STRING
    groupId: kafkaConsumerGroupId
Properties

The value to parse

Defaulttrue

If true, break on first match

The first successful match by grok will result in the task being finished. Set to false if you want the task to try all configured patterns.

Defaultfalse

If true, keep empty captures

When an optional field cannot be captured, the empty field is retained in the output. Set false if you want empty optional fields to be filtered out.

Defaulttrue

If true, only store named captures from grok

The Grok pattern to match

Custom pattern definitions

A map of pattern-name and pattern pairs defining custom patterns to be used by the current tasks. Patterns matching existing names will override the pre-existing definition.

SubTypestring

The list of Grok patterns to match

SubTypestring

List of user-defined pattern directories

Directories must be paths relative to the working directory.

Reference (ref) of the pluginDefaults to apply to this task.

More Plugins in this Category

Clickhouse
Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

5 tasks

2 blueprints

Weaviate
Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

4 tasks

2 blueprints