PutRecords
Send records to Amazon Kinesis Data Streams.
type: "io.kestra.plugin.aws.kinesis.PutRecords"Examples
Send multiple records as maps to Amazon Kinesis Data Streams. Check the following AWS API reference for the structure of the PutRecordsRequestEntry request payload.
id: aws_kinesis_put_records
namespace: company.team
tasks:
- id: put_records
type: io.kestra.plugin.aws.kinesis.PutRecords
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
streamName: "mystream"
records:
- data: "user sign-in event"
explicitHashKey: "optional hash value overriding the partition key"
partitionKey: "user1"
- data: "user sign-out event"
partitionKey: "user1"
Send multiple records from an internal storage ion file to Amazon Kinesis Data Streams.
id: aws_kinesis_put_records
namespace: company.team
tasks:
- id: put_records
type: io.kestra.plugin.aws.kinesis.PutRecords
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
streamName: "mystream"
records: kestra:///myfile.ion
Properties
records*Requiredstringarray
List of records (i.e., list of maps) or internal storage URI of the file that defines the records to be sent to AWS Kinesis Data Streams.
A list of at least one record with a map including data and partitionKey properties (those two are required arguments). Check the PutRecordsRequestEntry API reference for a detailed description of required fields.
accessKeyIdstring
Access Key Id in order to connect to AWS.
compatibilityModebooleanstring
Enable compatibility mode.
Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.
endpointOverridestring
The endpoint with which the SDK should communicate.
This property allows you to use a different S3 compatible storage backend.
failOnUnsuccessfulRecordsbooleanstring
trueMark the task as failed when sending a record is unsuccessful.
If true, the task will fail when any record fails to be sent.
forcePathStylebooleanstring
Force path style access.
Must only be used when compatibilityMode is enabled.
regionstring
AWS region with which the SDK should communicate.
secretKeyIdstring
Secret Key Id in order to connect to AWS.
sessionTokenstring
AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource.
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
streamArnstring
The ARN of the stream to push the records.
streamNamestring
The name of the stream to push the records.
Make sure to set either streamName or streamArn. One of those must be provided.
stsEndpointOverridestring
The AWS STS endpoint with which the SDKClient should communicate.
stsRoleArnstring
AWS STS Role.
The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
stsRoleExternalIdstring
AWS STS External Id.
A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.
stsRoleSessionDurationstring
PT15MdurationAWS STS Session duration.
The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.
stsRoleSessionNamestring
AWS STS Session name.
This property is only used when an stsRoleArn is defined.
Outputs
failedRecordsCountinteger
0The number of failed records.
recordCountinteger
0The total number of records sent to AWS Kinesis Data Streams.
uristring
uriThe URI of stored data
The successfully and unsuccessfully ingested records.If the ingestion was successful, the output includes the record sequence number.Otherwise, the output provides the error code and error message for troubleshooting.
Metrics
durationtimer
nanosecondsExecution time for the PutRecords task.
record.countcounter
recordsTotal number of records attempted to be sent to Kinesis.
record.failedcounter
recordsNumber of records that failed to be sent to Kinesis.
record.successfulcounter
recordsNumber of records successfully sent to Kinesis.
