Apache Iggy Consume
CertifiedConsume messages from Apache Iggy stream and topic
Apache Iggy Consume
Consume messages from Apache Iggy stream and topic
Pulls messages from an Apache Iggy stream and topic, tracks consumer offsets, and stores consumed messages into Kestra internal storage. Designed for UTF-8 text and JSON payloads; arbitrary binary payloads are not supported.
type: io.kestra.plugin.iggy.ConsumeExamples
Consume messages from an Apache Iggy topic
id: iggy_consume
namespace: company.team
tasks:
- id: consume
type: io.kestra.plugin.iggy.Consume
host: iggy.example.com
port: 8090
username: "{{ secret('IGGY_USERNAME') }}"
password: "{{ secret('IGGY_PASSWORD') }}"
stream: orders
topic: created
maxRecords: 100
maxDuration: PT30S
- id: log_count
type: io.kestra.plugin.core.log.Log
message: "Consumed {{ outputs.consume.messagesCount }} messages"
Properties
host *string
Server host
Hostname or IP address of the Apache Iggy server.
stream *string
Stream identifier or name
Name or ID of the Apache Iggy stream to consume messages from.
topic *string
Topic identifier or name
Name or ID of the Apache Iggy topic to consume messages from.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150autoCommit booleanstring
trueAuto-commit offsets
Whether to automatically commit consumer offsets upon polling. Defaults to true.
batchSize integerstring
50Batch size
Maximum messages fetched per poll. Must be greater than or equal to 1. Defaults to 50.
consumerId string
1Consumer ID or Name
Identifier of the consumer used to track offsets. Defaults to '1'.
maxDuration string
Maximum polling duration
Maximum duration to poll before stopping execution.
maxRecords integerstring
Maximum records to consume
Optional cap on total messages before stopping.
offset integerstring
Offset value
Offset or timestamp value to use when strategy is Offset or Timestamp.
partitionId integerstring
Partition ID
Optional partition ID to consume from.
password string
Authentication password
Password for authentication.
pollDuration string
PT1SPoll interval duration
Wait duration between empty poll attempts. Defaults to PT1S.
port integerstring
8090Server port
Port of the Apache Iggy server. Defaults to 8090 for TCP and 3000 for HTTP.
strategy string
NextOffsetTimestampFirstLastNextPolling strategy kind
Polling strategy: Offset, Timestamp, First, Last, or Next. Defaults to Next.
tls booleanstring
falseUse TLS
Whether to enable TLS encryption for the connection. Defaults to false.
token string
Personal access token
Personal access token (PAT) for token-based authentication.
transport string
TCPTCPHTTPTransport protocol
Transport protocol to use: TCP (default) or HTTP.
username string
Authentication username
Username for authentication.
Outputs
messagesCount integer
Messages consumed count
Total number of messages consumed during this task execution.
uri string
uriOutput file URI
Kestra internal storage URI of the file containing consumed messages.
