Huawei CreateClusterAndSubmitJob

Huawei CreateClusterAndSubmitJob
Certified
Create a Huawei Cloud MRS cluster and submit job steps to it

Creates a new MRS (MapReduce Service) cluster and, optionally, submits one or more job steps to run on it once it is ready — the Huawei Cloud equivalent of io.kestra.plugin.aws.emr.CreateClusterAndSubmitSteps.

MRS cluster creation requires a richer set of properties than EMR: VPC/subnet placement, node group sizing per role, and either a manager admin password or a login key pair. Billing is always pay-per-use (postPaid) — prepaid clusters are not supported by this task.

When steps is omitted, the cluster is created via MRS's createCluster API (cluster-only — that endpoint has no steps field). Jobs can then be submitted to the running cluster with the SubmitJob task. When steps is provided, the cluster and its steps are created together via MRS's runJobFlow API, which requires a non-empty steps field.

Set wait to true (the default) to poll until the cluster reaches the running state, at which point step job IDs (if any steps were submitted) are best-effort resolved and returned. Set wait to false to return immediately after the cluster creation request is accepted — in that case clusterState and jobIds are left empty, since the cluster and its steps are still being provisioned asynchronously.

The created cluster is a long-lived infrastructure resource: killing this task does not delete it. Use DeleteCluster to tear it down.

yaml
type: io.kestra.plugin.huawei.mrs.CreateClusterAndSubmitJob

Examples

Create a 5-node analysis cluster (2 masters + 3 core) and submit a Spark job.

yaml
id: mrs_create_cluster_and_submit_job
namespace: company.team

tasks:
  - id: create_cluster
    type: io.kestra.plugin.huawei.mrs.CreateClusterAndSubmitJob
    accessKeyId: "{{ secret('HUAWEI_AK') }}"
    secretAccessKey: "{{ secret('HUAWEI_SK') }}"
    region: eu-west-101
    projectId: "{{ secret('HUAWEI_PROJECT_ID') }}"
    clusterName: kestra-qa-cluster
    clusterVersion: "MRS 3.5.0-LTS" # current LTS -> in-stock, current-gen node flavors
    clusterType: ANALYSIS
    availabilityZone: eu-west-101a # AZ id, not the region itself
    vpcName: vpc-default
    subnetName: subnet-default
    components:
      - Hadoop
      - Spark # "Spark" on 3.5.x, NOT "Spark2x"
      - Hive
    nodeGroups:
      - groupName: master_node_default_group
        nodeNum: 2 # master group must have >= 2 nodes
        nodeSize: c6.4xlarge.4.linux.bigdata # note the .linux.bigdata suffix
        rootVolumeType: SAS
        rootVolumeSize: 480
        dataVolumeType: SAS
        dataVolumeSize: 600 # data disk must be >= 480 GB
        dataVolumeCount: 1
      - groupName: core_node_analysis_group # "analysis" core group for ANALYSIS clusters
        nodeNum: 3
        nodeSize: c6.4xlarge.4.linux.bigdata
        rootVolumeType: SAS
        rootVolumeSize: 480
        dataVolumeType: SAS
        dataVolumeSize: 600
        dataVolumeCount: 1
    loginMode: PASSWORD
    nodeRootPassword: "{{ secret('MRS_NODE_ROOT_PASSWORD') }}"
    managerAdminPassword: "{{ secret('MRS_MANAGER_ADMIN_PASSWORD') }}"
    steps:
      - jobType: SPARK_SUBMIT
        jobName: daily-aggregation
        arguments:
          - "--class"
          - "com.example.DailyAggregation"
          - "obs://my-bucket/jars/etl.jar"
    maxDuration: PT30M

Properties

availabilityZone *string

Availability zone

E.g. eu-west-101a.

clusterName *string

Cluster name

clusterType *string

Possible Values

ANALYSISSTREAMINGMIXEDCUSTOM

Cluster type

clusterVersion *string

MRS runtime version

E.g. MRS 3.2.0-LTS.3. See the MRS console for versions available in your region.

components *array

SubTypestring

Hadoop ecosystem components to install

E.g. Hadoop, Spark2x, Hive, Tez, Flink.

loginMode *string

Possible Values

PASSWORDPUBLICKEY

Node SSH login method

managerAdminPassword *string

MRS Manager web console admin password

Required regardless of loginMode.

nodeGroups *array

Node groups

At least a master and a core node group are typically required — see the MRS console for the groups a given clusterVersion/clusterType expects.

Definitions

io.kestra.plugin.huawei.mrs.models.NodeGroupConfig

groupName*string

Node group name

The role of this node group, e.g. master_node_default_group, core_node_default_group, or task_node_default_group — must match a group name accepted by the chosen clusterVersion/clusterType.

nodeNum*integerstring

Number of nodes in this group

Must be >= 1.

nodeSize*string

ECS flavor for nodes in this group

E.g. c6.4xlarge.4. See the MRS console for flavors available in your region.

dataVolumeCountintegerstring

Default1

Number of data volumes per node

Must be between 1 and 20. Defaults to 1.

dataVolumeSizeintegerstring

Default100

Data volume size (GB)

Must be >= 1. Defaults to 100 GB.

dataVolumeTypestring

DefaultSATA

Data volume type

Defaults to SATA.

rootVolumeSizeintegerstring

Default40

Root volume size (GB)

Must be >= 1. Defaults to 40 GB.

rootVolumeTypestring

DefaultSATA

Root volume type

Defaults to SATA.

vpcName *string

VPC name

accessKeyId string

Access Key (AK) used to authenticate with Huawei Cloud

Huawei Cloud access key used together with secretAccessKey to sign API requests. Required for AK/SK-based authentication; not required when providing a pre-obtained securityToken. Sensitive — always provide via {{ secret('NAME') }}.

deleteWhenNoSteps booleanstring

Defaultfalse

Delete the cluster automatically once all submitted steps complete

Defaults to false (the cluster keeps running after its steps finish). Set to true for ephemeral, job-only clusters.

domainId string

Huawei Cloud Account Domain ID

Identifies the Huawei Cloud account (domain). Required when authenticating against global services such as IAM, or when requesting a domain-scoped IAM token.

endpointOverride string

MRS endpoint URL override

Overrides the default endpoint derived from region and endpointSuffix. Use this for private endpoints, non-standard deployments, or tests. When set, endpointSuffix is ignored.

Format: https://mrs.<region>.myhuaweicloud.com (without trailing slash).

endpointSuffix string

Huawei Cloud domain suffix

Controls the top-level domain used when deriving the MRS endpoint from region. Defaults to myhuaweicloud.com. Set to myhuaweicloud.eu for the Huawei Cloud European sovereign cloud.

Ignored when endpointOverride is set.

enterpriseProjectId string

Enterprise project ID

interval string

DefaultPT15S

Polling interval while waiting for the cluster

ISO-8601 duration. Defaults to 15 seconds.

logUri string

OBS URI to collect component logs to

E.g. obs://my-bucket/mrs-logs/.

maxDuration string

DefaultPT30M

Maximum time to wait for the cluster to reach running

ISO-8601 duration. Defaults to 30 minutes.

nodeKeypairName string

Node login key pair name

Required when loginMode is PUBLICKEY. Must already exist in the target region.

nodeRootPassword string

Node root password

Required when loginMode is PASSWORD.

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

projectId string

Huawei Cloud Project ID

Identifies the region-scoped project against which most regional services authenticate. Mutually exclusive with domainId for global services such as IAM.

region string

Huawei Cloud region

Region identifier such as eu-west-101, ap-southeast-1, or cn-north-4.

safeMode string

DefaultSIMPLE

Possible Values

SIMPLEKERBEROS

Cluster authentication mode

Defaults to SIMPLE.

secretAccessKey string

Secret Key (SK) used to authenticate with Huawei Cloud

Huawei Cloud secret key paired with accessKeyId. Required for AK/SK-based authentication. Sensitive — always provide via {{ secret('NAME') }}.

securityGroupsId string

Security group ID

When omitted, MRS creates a default security group for the cluster automatically.

securityToken string

Pre-obtained Huawei Cloud IAM token used as bearer credential for downstream API calls

When set, downstream Huawei tasks send this value in the X-Auth-Token header instead of signing requests with AK/SK. Sensitive.

steps array

Job steps to submit once the cluster is ready

Each step is submitted individually via MRS's job-execution API as soon as the cluster reaches the running state. Optional: when omitted, the task only creates the cluster (no jobs are queued) — use the SubmitJob task to submit jobs afterward.

Definitions

io.kestra.plugin.huawei.mrs.models.JobConfig

jobName*string

Job name

A unique, human-readable name for this job step within the cluster.

jobType*string

Possible Values

MAP_REDUCESPARK_SUBMITHIVE_SCRIPTHIVE_SQLDIST_CPSPARK_SCRIPTSPARK_SQLFLINK

Job execution type

argumentsarray

SubTypestring

Job arguments

Positional arguments passed to the job, e.g. the main class, jar path, or script arguments.

propertiesobject

Extra job configuration entries

Key/value pairs forwarded as-is to the job's properties field.

subnetId string

Subnet ID

Either subnetId or subnetName is required.

subnetName string

Subnet name

Either subnetId or subnetName is required.

temporaryCredentials string

Inline IAM credential exchange

When set, the connection layer calls the Huawei IAM STS API once per task execution and uses the returned temporary AK/SK + security token instead of the static accessKeyId and secretAccessKey properties.

Configure once via pluginDefaults to apply transparently to every task in a namespace without per-task credential wiring:

pluginDefaults: 
  - type: io.kestra.plugin.huawei.obs
    values: 
      region: eu-west-101
      temporaryCredentials: 
        authMethod: PASSWORD
        username: my-iam-user
        password: "{{ secret('HUAWEI_IAM_PASSWORD') }}"
        domainName: my-account-domain
        durationSeconds: 3600

**Long-running tasks: ** the exchange runs once at execution start. For RealtimeTrigger or long-running Consume tasks that outlive durationSeconds, credentials will expire mid-run. Use long-lived AK/SK properties or refresh externally in that case.

Definitions

io.kestra.plugin.huawei.TemporaryCredentialsConfig

authMethodstring

DefaultPASSWORD

Possible Values

PASSWORDTOKEN

Authentication method

Controls which credentials are used to obtain the session token before exchanging for temporary STS credentials.

PASSWORD (default): provide username, password, and domainName.
TOKEN: provide an existing iamToken (X-Auth-Token).

domainNamestring

Account domain name (PASSWORD method only)

The Huawei Cloud account name (domain name) that owns the IAM user. Required when authMethod is PASSWORD. Visible in the Huawei Cloud console under My Credentials → Domain Name.

durationSecondsintegerstring

Default900

Lifetime of the temporary credentials in seconds

How long the returned temporary AK/SK/security-token should remain valid. Huawei Cloud accepts values between 900 (15 minutes) and 86400 (24 hours). Defaults to 900 seconds.

endpointSuffixstring

Defaultmyhuaweicloud.com

Huawei Cloud IAM endpoint suffix

Domain suffix used to build the IAM endpoint URL when no explicit endpoint override is set. Defaults to myhuaweicloud.com. Set to myhuaweicloud.eu for the European sovereign cloud (region eu-west-101 / EU-Dublin).

iamTokenstring

IAM token to exchange (TOKEN method only)

An existing Huawei Cloud X-Auth-Token to exchange for temporary STS credentials. Required when authMethod is TOKEN. Sensitive — always provide via {{ secret('NAME') }}.

passwordstring

IAM password (PASSWORD method only)

Password for the IAM user identified by username. Required when authMethod is PASSWORD. Sensitive — always provide via {{ secret('NAME') }}.

projectNamestring

Project name for project-scoped tokens (PASSWORD method only)

Overrides the project name used for scope=PROJECT token requests. Defaults to the task's region value when omitted, which is correct for most regions.

scopestring

DefaultPROJECT

Possible Values

PROJECTDOMAIN

Token scope (PASSWORD method only)

Scope of the session token obtained during password authentication.

PROJECT (default): token is scoped to the project matching projectName (or the task's region when projectName is omitted). Use for most downstream tasks.
DOMAIN: token is scoped to the domain.

usernamestring

IAM username (PASSWORD method only)

Huawei Cloud IAM username. Required when authMethod is PASSWORD.

wait booleanstring

Defaulttrue

Wait for the cluster to reach the running state

When true (the default), polls cluster status until running (or a failure state) and best-effort resolves step job IDs once the cluster is ready. When false, returns immediately after the creation request is accepted, leaving clusterState and jobIds empty.

Outputs

clusterId string

ID of the created cluster

clusterState string

Cluster state when the task returned

running on success. null when wait is false.