PutRecords
type: "io.kestra.plugin.aws.kinesis.PutRecords"
Send multiple records to Amazon Kinesis Data Streams.
Examples
Send multiple records as maps to Amazon Kinesis Data Streams. Check the following AWS API reference for the structure of the PutRecordsRequestEntry request payload.
id: "put_records"
type: "io.kestra.plugin.aws.kinesis.PutRecords"
streamName: "mystream"
records:
- data: "user sign-in event"
explicitHashKey: "optional hash value overriding the partition key"
- data: "user sign-out event"
partitionKey: "user1"
Send multiple records from an internal storage ion file to Amazon Kinesis Data Streams.
id: "put_records"
type: "io.kestra.plugin.aws.kinesis.PutRecords"
streamName: "mystream"
records: kestra://myfile.ion
Properties
failOnUnsuccessfulRecords
- Type: boolean
- Dynamic: ❌
- Required: ✔️
- Default:
true
Mark the task as failed when sending a record is unsuccessful.
If true, the task will fail when any record fails to be sent.
records
- Type:stringarray
- Dynamic: ✔️
- Required: ✔️
List of records (i.e., list of maps) or internal storage URI of the file that defines the records to be sent to AWS Kinesis Data Streams.
A list of at least one record with a map including data
and partitionKey
properties (those two are required arguments). Check the PutRecordsRequestEntry API reference for a detailed description of required fields.
accessKeyId
- Type: string
- Dynamic: ✔️
- Required: ❌
The Access Key Id in order to connect to AWS
If no connection is defined, we will use default DefaultCredentialsProvider that will try to guess the value
compatibilityMode
- Type: boolean
- Dynamic: ❓
- Required: ❌
endpointOverride
- Type: string
- Dynamic: ✔️
- Required: ❌
The endpoint with which the SDK should communicate.
Should normally not be used except for local development.
region
- Type: string
- Dynamic: ✔️
- Required: ❌
The region with which the SDK should communicate.
secretKeyId
- Type: string
- Dynamic: ✔️
- Required: ❌
The Secret Key Id in order to connect to AWS
If no connection is defined, we will use default DefaultCredentialsProvider that will try to guess the value
sessionToken
- Type: string
- Dynamic: ✔️
- Required: ❌
The he AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permission to access some resource.
If no connection is defined, we will use default DefaultCredentialsProvider that will try to guess the value
streamArn
- Type: string
- Dynamic: ✔️
- Required: ❌
The ARN of the stream to add the records.
Make sure to set either streamName
or streamArn
. One of those must be provided.
streamName
- Type: string
- Dynamic: ✔️
- Required: ❌
The name of the stream to add the records.
Make sure to set either streamName
or streamArn
. One of those must be provided.
Outputs
failedRecordsCount
- Type: integer
- Default:
0
The number of failed records.
recordCount
- Type: integer
- Default:
0
The total number of records sent to AWS Kinesis Data Streams.
uri
- Type: string
The URI of stored data
The successfully and unsuccessfully ingested records.If the ingestion was successful, the output includes the record sequence number.Otherwise, the output provides the error code and error message for troubleshooting.
Definitions
Record
data
- Type: string
- Dynamic: ✔️
- Required: ✔️
Free-form data blob to put into the record.
partitionKey
- Type: string
- Dynamic: ✔️
- Required: ✔️
Determines which shard in the stream the data record is assigned to.
explicitHashKey
- Type: string
- Dynamic: ✔️
- Required: ❌
The optional hash value used to determine explicitly the shard that the data record is assigned to by overriding the partition key hash.