Apache Kafka AwsGlueSchemaRegistry

Apache Kafka AwsGlueSchemaRegistry

Certified

AWS Glue Schema Registry

yaml
type: io.kestra.plugin.kafka.registry.AwsGlueSchemaRegistry

Consume AVRO data from a Kafka topic using AWS Glue Schema Registry

yaml
id: consume-kafka-messages
namespace: company.team

tasks:
  - id: consume
    type: io.kestra.plugin.kafka.Consume
    topic: topic_test
    properties:
      bootstrap.servers: localhost:9093
      auto.offset.reset: earliest
    pollDuration: PT20S
    maxRecords: 50
    keyDeserializer: STRING
    valueDeserializer: AVRO
    schemaRegistryVendor:
      type: io.kestra.plugin.kafka.registry.AwsGlueSchemaRegistry
      region: us-east-1
      # endpoint, accessKey, secretKey are optional when using IAM roles
      accessKey: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKey: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-kafka --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

The AWS region where the schema registry is located

The AWS access key ID. Optional when using IAM roles or environment credentials

The AWS Glue Schema Registry endpoint. Optional when using IAM roles or standard regional endpoints

The AWS secret access key. Optional when using IAM roles or environment credentials