Huawei Consume

Huawei Consume
Certified
Consume records from a Huawei Cloud DIS (Data Ingestion Service) stream

Reads records from every partition of the configured stream (or a single partitionId, when set) until maxRecords or maxDuration is reached (at least one is required), or until the stream is fully caught up — whichever comes first. Records are written to Kestra internal storage as ION at uri.

This task always starts reading from startingPosition (or startingTimestamp) — it does not remember where a previous execution left off. For continuous, non-overlapping consumption use Trigger or RealtimeTrigger, which persist a per-partition sequence-number watermark.

yaml
type: io.kestra.plugin.huawei.dis.Consume

Examples

yaml
id: dis_consume
namespace: company.team

tasks:
  - id: consume
    type: io.kestra.plugin.huawei.dis.Consume
    accessKeyId: "{{ secret('HUAWEI_AK') }}"
    secretAccessKey: "{{ secret('HUAWEI_SK') }}"
    region: eu-west-101
    streamName: my-stream
    maxRecords: 500

Properties

streamName *string

DIS stream name

accessKeyId string

Access Key (AK) used to authenticate with Huawei Cloud

Huawei Cloud access key used together with secretAccessKey to sign API requests. Required for AK/SK-based authentication; not required when providing a pre-obtained securityToken. Sensitive — always provide via {{ secret('NAME') }}.

domainId string

Huawei Cloud Account Domain ID

Identifies the Huawei Cloud account (domain). Required when authenticating against global services such as IAM, or when requesting a domain-scoped IAM token.

endpointOverride string

DIS endpoint URL override

Overrides the default endpoint derived from region and endpointSuffix. Use this for private endpoints, non-standard deployments, or tests. When set, endpointSuffix is ignored.

Format: https://dis.<region>.myhuaweicloud.com (without trailing slash).

endpointSuffix string

Huawei Cloud domain suffix

Controls the top-level domain used when deriving the DIS endpoint from region. Defaults to myhuaweicloud.com. Set to myhuaweicloud.eu for the Huawei Cloud European sovereign cloud.

Ignored when endpointOverride is set.

maxDuration string

Stop after this duration has elapsed

ISO-8601 duration, e.g. PT30S. Maximum PT24H. At least one of maxRecords or maxDuration must be set.

maxFetchBytes integerstring

Default2097152

Maximum bytes fetched per partition, per call

Defaults to 2 MB (2097152 bytes), DIS's per-request payload ceiling.

maxRecords integerstring

Stop after consuming this many records

Upper bound across all partitions combined. Maximum 1000000. At least one of maxRecords or maxDuration must be set.

partitionId string

Single partition to consume from

When omitted (default), every partition of the stream is consumed.

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

projectId string

Huawei Cloud Project ID

Identifies the region-scoped project against which most regional services authenticate. Mutually exclusive with domainId for global services such as IAM.

region string

Huawei Cloud region

Region identifier such as eu-west-101, ap-southeast-1, or cn-north-4.

secretAccessKey string

Secret Key (SK) used to authenticate with Huawei Cloud

Huawei Cloud secret key paired with accessKeyId. Required for AK/SK-based authentication. Sensitive — always provide via {{ secret('NAME') }}.

securityToken string

Pre-obtained Huawei Cloud IAM token used as bearer credential for downstream API calls

When set, downstream Huawei tasks send this value in the X-Auth-Token header instead of signing requests with AK/SK. Sensitive.

serdeType string

DefaultSTRING

Possible Values

STRINGJSONBINARY

Deserialization type applied to each record's data value

startingPosition string

DefaultTRIM_HORIZON

Possible Values

TRIM_HORIZONLATESTAT_TIMESTAMP

Where to start reading a partition when no earlier position is known

TRIM_HORIZON (default) reads from the oldest retained record. LATEST reads only new records. AT_TIMESTAMP reads from the oldest record at or after startingTimestamp (required in that case). Triggers apply this only the first time a partition has no persisted watermark yet.

startingTimestamp string

Starting timestamp

Required when startingPosition is AT_TIMESTAMP.

temporaryCredentials string

Inline IAM credential exchange

When set, the connection layer calls the Huawei IAM STS API once per task execution and uses the returned temporary AK/SK + security token instead of the static accessKeyId and secretAccessKey properties.

Configure once via pluginDefaults to apply transparently to every task in a namespace without per-task credential wiring:

pluginDefaults: 
  - type: io.kestra.plugin.huawei.obs
    values: 
      region: eu-west-101
      temporaryCredentials: 
        authMethod: PASSWORD
        username: my-iam-user
        password: "{{ secret('HUAWEI_IAM_PASSWORD') }}"
        domainName: my-account-domain
        durationSeconds: 3600

**Long-running tasks: ** the exchange runs once at execution start. For RealtimeTrigger or long-running Consume tasks that outlive durationSeconds, credentials will expire mid-run. Use long-lived AK/SK properties or refresh externally in that case.

Definitions

io.kestra.plugin.huawei.TemporaryCredentialsConfig

authMethodstring

DefaultPASSWORD

Possible Values

PASSWORDTOKEN

Authentication method

Controls which credentials are used to obtain the session token before exchanging for temporary STS credentials.

PASSWORD (default): provide username, password, and domainName.
TOKEN: provide an existing iamToken (X-Auth-Token).

domainNamestring

Account domain name (PASSWORD method only)

The Huawei Cloud account name (domain name) that owns the IAM user. Required when authMethod is PASSWORD. Visible in the Huawei Cloud console under My Credentials → Domain Name.

durationSecondsintegerstring

Default900

Lifetime of the temporary credentials in seconds

How long the returned temporary AK/SK/security-token should remain valid. Huawei Cloud accepts values between 900 (15 minutes) and 86400 (24 hours). Defaults to 900 seconds.

endpointSuffixstring

Defaultmyhuaweicloud.com

Huawei Cloud IAM endpoint suffix

Domain suffix used to build the IAM endpoint URL when no explicit endpoint override is set. Defaults to myhuaweicloud.com. Set to myhuaweicloud.eu for the European sovereign cloud (region eu-west-101 / EU-Dublin).

iamTokenstring

IAM token to exchange (TOKEN method only)

An existing Huawei Cloud X-Auth-Token to exchange for temporary STS credentials. Required when authMethod is TOKEN. Sensitive — always provide via {{ secret('NAME') }}.

passwordstring

IAM password (PASSWORD method only)

Password for the IAM user identified by username. Required when authMethod is PASSWORD. Sensitive — always provide via {{ secret('NAME') }}.

projectNamestring

Project name for project-scoped tokens (PASSWORD method only)

Overrides the project name used for scope=PROJECT token requests. Defaults to the task's region value when omitted, which is correct for most regions.

scopestring

DefaultPROJECT

Possible Values

PROJECTDOMAIN

Token scope (PASSWORD method only)

Scope of the session token obtained during password authentication.

PROJECT (default): token is scoped to the project matching projectName (or the task's region when projectName is omitted). Use for most downstream tasks.
DOMAIN: token is scoped to the domain.