Stream JSON and Application Events to AWS Kinesis Data Streams at Scale

Push high-volume application events and JSON payloads into AWS Kinesis Data Streams with Kestra, using either inline record definitions or data fetched from an HTTP API. This blueprint solves a recurring real-time ingestion problem: getting events from many sources into a Kinesis stream reliably and in bulk, without writing custom producer code or wiring up a standalone scheduler. It covers both ad-hoc event publishing and an API-to-stream pipeline, with the JSON-to-ION conversion that the Kinesis PutRecords API requires handled for you.

How it works

1. put_records_simple_map (io.kestra.plugin.aws.kinesis.PutRecords) writes two inline records (sign-in and sign-out) to the kestra stream, each tagged with a partitionKey, showing the simplest way to emit events.
2. extract (io.kestra.plugin.core.http.Download) fetches a JSON dataset of user events from a public HTTP endpoint.
3. json_to_ion (io.kestra.plugin.serdes.json.JsonToIon) converts the downloaded JSON into the Amazon ION format expected by Kinesis, with newLine: false.
4. put_records (io.kestra.plugin.aws.kinesis.PutRecords) publishes the converted records to the kestra stream in a single bulk call, referencing {{ outputs.json_to_ion.uri }}.

What you get

• A working two-pattern producer: inline events and API-driven bulk ingestion.
• Automatic JSON-to-ION conversion, so payloads match what PutRecords accepts.
• Per-record shard ID and sequence number in the Outputs tab, ready for later retrieval via the AWS GetRecords API.
• A declarative, version-controllable alternative to bespoke producer scripts.

Who it's for

• Data engineers building real-time ingestion and analytics pipelines on AWS.
• Platform teams standardizing how services emit events to Kinesis.
• Developers prototyping event-driven architectures without boilerplate producer code.

Why orchestrate this with Kestra

Kinesis ingests records but does not schedule or coordinate the producers that feed it. Kestra fills that gap: define the whole pipeline as declarative YAML, attach event or schedule triggers to drive ingestion, add retries so transient API or stream errors do not drop data, and keep full execution lineage across the download, conversion, and publish steps. You get observability and replayability that a raw PutRecords call cannot offer on its own.

Prerequisites

• An AWS account with a Kinesis Data Stream named kestra (or change streamName).
• IAM permissions for kinesis:PutRecords on that stream.
• AWS credentials available to the worker (default credential chain, instance role, or plugin defaults).

Secrets

This flow does not reference any {{ secret('...') }} values; it relies on the AWS default credential chain. To use explicit credentials, add accessKeyId, secretKeyId, and region to the PutRecords tasks (or pluginDefaults) and store them with {{ secret('AWS_ACCESS_KEY_ID') }} and {{ secret('AWS_SECRET_KEY_ID') }}.

Quick start

1. Create or select a Kinesis stream and set streamName on both PutRecords tasks.
2. Make AWS credentials reachable by the Kestra worker.
3. Add this blueprint to your namespace and run it.
4. Open the Outputs tab to confirm the returned shard IDs and sequence numbers.

How to extend

• Swap the extract URL for your own API or replace it with a file or database source.
• Add a schedule or webhook trigger to ingest events continuously.
• Set per-record partitionKey values to control shard distribution.
• Chain a downstream consumer or analytics task after publishing.

Note: the data field sent to Kinesis must be a string. Wrap structured JSON objects as strings before publishing.

Links

• AWS plugin documentation
• AWS orchestration with Kestra