Apache Kafka TopicDescribe

Apache Kafka TopicDescribe
Certified
Describe a Kafka topic

Returns a topic's partition layout (leader, replicas, in-sync replicas) and its effective configuration. Set includeOffsets: true to also return the earliest and latest offset of each partition. Fails with UnknownTopicOrPartitionException if the topic does not exist.

yaml
type: io.kestra.plugin.kafka.TopicDescribe

Examples

Inspect a tenant topic's retention config

yaml
id: kafka_topic_describe
namespace: company.team

tasks:
  - id: topic_describe
    type: io.kestra.plugin.kafka.TopicDescribe
    properties:
      bootstrap.servers: localhost:9092
    topic: tenant_acme_orders

  - id: log_retention
    type: io.kestra.plugin.core.log.Log
    message: "retention.ms = {{ outputs.topic_describe.configs['retention.ms'] }}"

Check the offset range retained in each partition

yaml
id: kafka_topic_offsets
namespace: company.team

tasks:
  - id: topic_describe
    type: io.kestra.plugin.kafka.TopicDescribe
    properties:
      bootstrap.servers: localhost:9092
    topic: tenant_acme_orders
    includeOffsets: true

  - id: log_offsets
    type: io.kestra.plugin.core.log.Log
    message: |
      {% for p in outputs.topic_describe.partitions %}
      partition {{ p.partition }}: offsets {{ p.earliestOffset }} to {{ p.latestOffset }}
      {% endfor %}

Properties

properties *object

Kafka AdminClient properties

Must include bootstrap.servers; accepts any Kafka AdminClient config. Provide base64-encoded content for ssl.keystore.location and ssl.truststore.location when using SSL.

topic *string

Topic name

assets

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions

io.kestra.core.models.assets.AssetsDeclaration

assetFailureBehaviorstring

Possible Values

IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

io.kestra.core.models.assets.AssetIdentifier

id*string

Min length1

typestring

outputs

The assets produced as outputs.

io.kestra.plugin.ee.assets.Dataset

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.File

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.Table

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.VM

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.core.models.assets.External

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.core.models.assets.Custom

id*string

Min length1

Max length150

type*string

Min length1

Custom asset type

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

callTimeout string

DefaultPT30S

AdminClient call timeout

Maximum duration to wait for each AdminClient operation to complete before failing the task. Defaults to PT30S (30 seconds). Distinct from the task-level timeout, which lets the worker kill the task without retrying it.

includeOffsets booleanstring

Defaultfalse

Include partition offsets

When true, each partition entry also contains earliestOffset (log start offset) and latestOffset (high watermark, the offset of the next record a consumer can read), fetched with two batched listOffsets requests. Transaction markers take offsets too, so latestOffset - earliestOffset can be larger than the record count. Both are null for a partition without a leader. Defaults to false, with no extra broker request and no offset keys.

Outputs

configs object

SubTypestring

Effective topic configuration

For example retention.ms, cleanup.policy.

partitionCount integer

Number of partitions

partitions array

SubTypeobject

Per-partition layout

Each entry contains partition, leader, replicas and isr (in-sync replicas broker IDs). With includeOffsets: true it also contains earliestOffset and latestOffset.