AWS CreateCluster

AWS CreateCluster
Certified
Create an Amazon MSK cluster

Provisions a new MSK cluster and returns its ARN and initial state. Cluster creation is asynchronous — use DescribeCluster or the Trigger task to poll for ACTIVE state before connecting Kafka producers or consumers. Note: this task creates a cluster with default encryption settings. For production workloads requiring encryption-in-transit or at-rest, configure those settings via the AWS Console or CLI after cluster creation.

yaml
type: io.kestra.plugin.aws.msk.CreateCluster

Examples

Create an MSK cluster and retrieve bootstrap brokers.

yaml
id: provision_msk_cluster
namespace: company.team

tasks:
  - id: create_cluster
    type: io.kestra.plugin.aws.msk.CreateCluster
    region: "{{ secret('AWS_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
    clusterName: "my-kestra-msk"
    kafkaVersion: "3.5.1"
    numberOfBrokerNodes: 3
    instanceType: "kafka.m5.large"
    clientSubnets:
      - "{{ secret('SUBNET_A') }}"
      - "{{ secret('SUBNET_B') }}"
      - "{{ secret('SUBNET_C') }}"
    securityGroups:
      - "{{ secret('SECURITY_GROUP_ID') }}"

  - id: log_arn
    type: io.kestra.plugin.core.log.Log
    message: "MSK cluster ARN: {{ outputs.create_cluster.clusterArn }}"

Properties

clientSubnets *array

SubTypestring

Client subnets

List of subnet IDs for broker node placement — one per Availability Zone.

clusterName *string

Cluster name

A unique name for the MSK cluster.

instanceType *string

Instance type

Broker instance type, e.g. kafka.m5.large.

kafkaVersion *string

Kafka version

The Apache Kafka version for the cluster, e.g. 3.5.1.

numberOfBrokerNodes *integerstring

Number of broker nodes

Total number of broker nodes. Must be a multiple of the number of Availability Zones.

securityGroups *array

SubTypestring

Security groups

List of security group IDs to associate with the broker nodes.

accessKeyId string

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

compatibilityMode booleanstring

Enable compatibility mode

Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.

ebsVolumeSize integerstring

EBS volume size (GiB)

Storage volume size per broker in GiB. Defaults to 100.

endpointOverride string

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

forcePathStyle booleanstring

Force path style access

Must only be used when compatibilityMode is enabled.

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

region string

AWS region with which the SDK should communicate

secretKeyId string

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

sessionToken string

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsEndpointOverride string

The AWS STS endpoint with which the SDKClient should communicate

stsRoleArn string

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalId string

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDuration string

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionName string

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Outputs

clusterArn string

Cluster ARN

The Amazon Resource Name of the newly created cluster.

clusterName string

Cluster name

The name of the cluster.

state string

State

Initial state of the cluster, typically CREATING.

NetBox

Enterprise

Enterprise NetBox tasks to create, update, and query devices, IP addresses, and sites over the NetBox API.

Infrastructure

Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

AWS CreateClusterCertifiedCreate an Amazon MSK cluster

AWS CreateCluster

More Plugins in this Category

NetBox

Clickhouse

AWS CreateCluster
Certified
Create an Amazon MSK cluster