Huawei Produce

Huawei Produce

Certified

Publish messages to a Huawei DMS for Kafka topic

Reads messages from from and sends them to the configured topic using the standard Apache Kafka protocol. Supports STRING, JSON, and BINARY serializers for both key and value. Each input map may contain key, value, topic (overrides the task-level topic), partition, and headers. All sends flush before the task exits.

yaml
type: io.kestra.plugin.huawei.dms.kafka.Produce
yaml
id: dms_kafka_produce
namespace: company.team

tasks:
  - id: produce
    type: io.kestra.plugin.huawei.dms.kafka.Produce
    bootstrapServers: "dms-instance-id.kafka.eu-west-101.myhuaweicloud.com:9093"
    saslMechanism: PLAIN
    username: "{{ secret('DMS_KAFKA_USERNAME') }}"
    password: "{{ secret('DMS_KAFKA_PASSWORD') }}"
    topic: my-topic
    from:
      key: "order-123"
      value: "Hello from Kestra!"

Produce JSON messages from a file

yaml
id: dms_kafka_produce_json
namespace: company.team

inputs:
  - id: dataFile
    type: FILE

tasks:
  - id: produce
    type: io.kestra.plugin.huawei.dms.kafka.Produce
    bootstrapServers: "dms-instance-id.kafka.eu-west-101.myhuaweicloud.com:9093"
    saslMechanism: PLAIN
    username: "{{ secret('DMS_KAFKA_USERNAME') }}"
    password: "{{ secret('DMS_KAFKA_PASSWORD') }}"
    topic: orders
    valueSerdeType: JSON
    from: "{{ inputs.dataFile }}"
Properties

Kafka bootstrap servers

Comma-separated list of host: port pairs that the Kafka client uses for the initial cluster connection. For DMS for Kafka, copy this value from the instance detail page in the console.

Messages to produce

A single map, a list of maps, or a URI pointing to an ION file in Kestra internal storage. Each map may contain: key, value, topic (overrides the task-level topic), partition, and headers.

Access Key (AK) used to authenticate with Huawei Cloud

Huawei Cloud access key used together with secretAccessKey to sign API requests. Required for AK/SK-based authentication; not required when providing a pre-obtained securityToken. Sensitive — always provide via {{ secret('NAME') }}.

Huawei Cloud Account Domain ID

Identifies the Huawei Cloud account (domain). Required when authenticating against global services such as IAM, or when requesting a domain-scoped IAM token.

DefaultSTRING
Possible Values
STRINGJSONBINARY

Key serializer/deserializer type

STRING (default), JSON, or BINARY.

SASL password

Required when saslMechanism is PLAIN or SCRAM_SHA_512. Sensitive — always provide via {{ secret('NAME') }}.

Reference (ref) of the pluginDefaults to apply to this task.

Huawei Cloud Project ID

Identifies the region-scoped project against which most regional services authenticate. Mutually exclusive with domainId for global services such as IAM.

Huawei Cloud region

Region identifier such as eu-west-101, ap-southeast-1, or cn-north-4.

DefaultPLAIN
Possible Values
PLAINSCRAM_SHA_512NONE

SASL mechanism used for authentication

PLAIN — username/password (default, used by most DMS for Kafka instances). SCRAM_SHA_512 — stronger challenge-response, supported on newer instances. NONE — no SASL; for VPC-internal clusters with no auth enabled.

Secret Key (SK) used to authenticate with Huawei Cloud

Huawei Cloud secret key paired with accessKeyId. Required for AK/SK-based authentication. Sensitive — always provide via {{ secret('NAME') }}.

Pre-obtained Huawei Cloud IAM token used as bearer credential for downstream API calls

When set, downstream Huawei tasks send this value in the X-Auth-Token header instead of signing requests with AK/SK. Sensitive.

Defaultfalse

Enable TLS for the Kafka connection

Set to true to use SASL_SSL instead of SASL_PLAINTEXT. DMS for Kafka instances accessed over the public internet require TLS.

Inline IAM credential exchange

When set, the connection layer calls the Huawei IAM STS API once per task execution and uses the returned temporary AK/SK + security token instead of the static accessKeyId and secretAccessKey properties.

Configure once via pluginDefaults to apply transparently to every task in a namespace without per-task credential wiring: 

pluginDefaults: 
  - type: io.kestra.plugin.huawei.obs
    values: 
      region: eu-west-101
      temporaryCredentials: 
        authMethod: PASSWORD
        username: my-iam-user
        password: "{{ secret('HUAWEI_IAM_PASSWORD') }}"
        domainName: my-account-domain
        durationSeconds: 3600

**Long-running tasks: ** the exchange runs once at execution start. For RealtimeTrigger or long-running Consume tasks that outlive durationSeconds, credentials will expire mid-run. Use long-lived AK/SK properties or refresh externally in that case.

Definitions
authMethodstring
DefaultPASSWORD
Possible Values
PASSWORDTOKEN

Authentication method

Controls which credentials are used to obtain the session token before exchanging for temporary STS credentials.

  • PASSWORD (default): provide username, password, and domainName.
  • TOKEN: provide an existing iamToken (X-Auth-Token).
domainNamestring

Account domain name (PASSWORD method only)

The Huawei Cloud account name (domain name) that owns the IAM user. Required when authMethod is PASSWORD. Visible in the Huawei Cloud console under My Credentials → Domain Name.

durationSecondsintegerstring
Default900

Lifetime of the temporary credentials in seconds

How long the returned temporary AK/SK/security-token should remain valid. Huawei Cloud accepts values between 900 (15 minutes) and 86400 (24 hours). Defaults to 900 seconds.

endpointSuffixstring
Defaultmyhuaweicloud.com

Huawei Cloud IAM endpoint suffix

Domain suffix used to build the IAM endpoint URL when no explicit endpoint override is set. Defaults to myhuaweicloud.com. Set to myhuaweicloud.eu for the European sovereign cloud (region eu-west-101 / EU-Dublin).

iamTokenstring

IAM token to exchange (TOKEN method only)

An existing Huawei Cloud X-Auth-Token to exchange for temporary STS credentials. Required when authMethod is TOKEN. Sensitive — always provide via {{ secret('NAME') }}.

passwordstring

IAM password (PASSWORD method only)

Password for the IAM user identified by username. Required when authMethod is PASSWORD. Sensitive — always provide via {{ secret('NAME') }}.

projectNamestring

Project name for project-scoped tokens (PASSWORD method only)

Overrides the project name used for scope=PROJECT token requests. Defaults to the task's region value when omitted, which is correct for most regions.

scopestring
DefaultPROJECT
Possible Values
PROJECTDOMAIN

Token scope (PASSWORD method only)

Scope of the session token obtained during password authentication.

  • PROJECT (default): token is scoped to the project matching projectName (or the task's region when projectName is omitted). Use for most downstream tasks.
  • DOMAIN: token is scoped to the domain.
usernamestring

IAM username (PASSWORD method only)

Huawei Cloud IAM username. Required when authMethod is PASSWORD.

Target Kafka topic

The topic to produce messages to. Can be overridden per record by setting topic inside from.

SASL username

Required when saslMechanism is PLAIN or SCRAM_SHA_512.

DefaultSTRING
Possible Values
STRINGJSONBINARY

Value serializer/deserializer type

STRING (default), JSON, or BINARY.

Number of records successfully sent to the DMS Kafka topic

Unitrecords

Number of records sent to the DMS Kafka topic.

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

OPA (Open Policy Agent)
OPA (Open Policy Agent)
Enterprise

Tasks to manage Open Policy Agent policies and decisions, including upload, evaluation, and partial evaluation.

Infrastructure

5 tasks

3 blueprints