AWS PutRecords

AWS PutRecords
Certified
Put records to Kinesis Data Streams

Sends multiple records to a stream by name or ARN. Records can be inline or loaded from a kestra:// ION file. Records require data and partitionKey; explicitHashKey is optional. Fails the task when any record fails if failOnUnsuccessfulRecords is true (default).

yaml
type: io.kestra.plugin.aws.kinesis.PutRecords

Examples

Send multiple records as maps to Amazon Kinesis Data Streams. Check the following AWS API reference for the structure of the PutRecordsRequestEntry request payload

yaml
id: aws_kinesis_put_records
namespace: company.team

tasks:
  - id: put_records
    type: io.kestra.plugin.aws.kinesis.PutRecords
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    streamName: "mystream"
    records:
      - data: "user sign-in event"
        explicitHashKey: "optional hash value overriding the partition key"
        partitionKey: "user1"
      - data: "user sign-out event"
        partitionKey: "user1"

Send multiple records from an internal storage ion file to Amazon Kinesis Data Streams

yaml
id: aws_kinesis_put_records
namespace: company.team

tasks:
  - id: put_records
    type: io.kestra.plugin.aws.kinesis.PutRecords
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    streamName: "mystream"
    records: kestra:///myfile.ion

Properties

records *stringarray

Records

List of records or kestra:// URI to an ION file of records. Each requires data and partitionKey; optional explicitHashKey.

accessKeyId string

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

compatibilityMode booleanstring

Enable compatibility mode

Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.

endpointOverride string

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

failOnUnsuccessfulRecords booleanstring

Defaulttrue

Fail on unsuccessful records

If true (default), task fails when at least one record is rejected.

forcePathStyle booleanstring

Force path style access

Must only be used when compatibilityMode is enabled.

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

region string

AWS region with which the SDK should communicate

secretKeyId string

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

sessionToken string

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

streamArn string

Stream ARN

Kinesis stream ARN; set this or streamName.

streamName string

Stream name

Kinesis stream name; set this or streamArn.

stsEndpointOverride string

The AWS STS endpoint with which the SDKClient should communicate

stsRoleArn string

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalId string

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDuration string

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionName string

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Outputs

failedRecordsCount integer

Default0

The number of failed records

recordCount integer

Default0

The total number of records sent to AWS Kinesis Data Streams

uri string

Formaturi

The URI of stored data

The successfully and unsuccessfully ingested records.If the ingestion was successful, the output includes the record sequence number.Otherwise, the output provides the error code and error message for troubleshooting.

Metrics

duration timer

Unitnanoseconds

Execution time for the PutRecords task.

record.count counter

Unitrecords

Total number of records attempted to be sent to Kinesis.

record.failed counter

Unitrecords

Number of records that failed to be sent to Kinesis.

record.successful counter

Unitrecords