Huawei DataArts Factory Job Run Health Gate

DataArts Studio's own Monitor Instance view shows a job's last run if someone opens it, but nothing pages anyone when a critical ETL job fails at 3am, gets stuck in waiting behind a cluster resource limit, or quietly stops firing because its schedule was disabled by mistake. This blueprint reads the single most recent run instance of a named job with io.kestra.plugin.huawei.dataarts.GetJobRun, classifies it, and alerts on anything that is not a clean, timely success.

This blueprint was created by Mohit Jeswani.

How it works

1. get_run (io.kestra.plugin.huawei.dataarts.GetJobRun) fetches the most recently created run instance for job_name. Read-only, no job is started or modified. allowFailure: true covers the one case the task itself throws on: a job with no run instance at all.
2. evaluate (io.kestra.plugin.core.output.OutputValues) classifies the result:
   • CRITICAL: status is fail, or get_run itself failed (no run ever found).
   • STUCK: status is running or waiting and the elapsed time since startTime exceeds max_run_minutes.
   • STALE: the latest instance's own planTime is older than max_staleness_hours, the signal that the job's schedule may have stopped firing rather than that this particular run is unhealthy.
   • OK: status is success.
   • NEEDS_ATTENTION: anything else (manual, pause, freeze, skip).
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs the healthy status.
   • alert: posts the severity, status, and instance identifiers to Slack.
4. summary logs the severity, status and instance ID, closing the governance cycle.
5. errors alerts Slack if the flow itself fails outside the handled cases, for example an expired AK/SK or an unreachable DataArts Studio endpoint.

What you get

• A single Slack alert that distinguishes an outright failure from a job stuck behind a resource limit from a schedule that silently stopped firing, three different problems a plain "last run failed" check cannot tell apart.
• A health check with zero side effects: it never starts, stops, or backfills the job it is watching, unlike using StartJobRun itself as the monitor.
• A record, in Kestra's own execution history, of every classification this job received over time.

Who it's for

• Data platform teams running ETL in Huawei Cloud DataArts Factory who want a critical job watched for failure and for staleness, not just failure.
• On-call engineers who want one Slack message naming the instance ID and status instead of opening the DataArts Studio console's Monitor Instance view to check.
• Teams evaluating the Huawei Cloud plugin who want a worked example of GetJobRun read alone, distinct from StartJobRun's own documented supplement-data workflow.

Why orchestrate this with Kestra

DataArts Factory has no native alerting on job staleness, and its own run history is a console view someone has to remember to open. Kestra supplies the schedule, keeps max_run_minutes and max_staleness_hours as flow inputs instead of buried configuration, and turns a silent status field into a Slack page with a severity attached, backed by a full history of every check.

Prerequisites

• A Huawei Cloud account with DataArts Studio enabled and at least one DataArts Factory job already created, with a cron schedule, HTTP trigger, or parent job configured (DataArts requires one of these before any instance can ever be created).
• An AK/SK pair (IAM access key and secret key) with DataArts Factory read permission.
• The Huawei Cloud project ID for the region the DataArts workspace lives in, and the workspace ID if the account has more than one workspace.
• A Slack incoming webhook URL.

This flow calls the Huawei Cloud DataArts Studio API directly over HTTPS; there is no official local DataArts emulator, so get_run cannot be exercised against a local Docker container. Point job_name at a real job with at least one run in a trial account to test it end to end.

Secrets

• HUAWEI_AK: the IAM access key used by get_run. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_HUAWEI_AK.
• HUAWEI_SK: the IAM secret key paired with HUAWEI_AK. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_HUAWEI_SK.
• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Type	Default	Purpose
region	STRING	eu-west-101	Huawei Cloud region used to sign DataArts requests
project_id	STRING	<your-huawei-project-id>	Required by every DataArts Factory API call
workspace_id	STRING	empty	DataArts Studio workspace, empty for the default one
job_name	STRING	my_etl_job	Job watched by this gate
max_run_minutes	INT	60	Elapsed running/waiting time reported as STUCK
max_staleness_hours	INT	26	Age of the latest run's planTime reported as STALE

Outputs

• {{ outputs.get_run.status }}: waiting, running, success, fail, manual, pause, skip or freeze. Present only when a run instance was found.
• {{ outputs.get_run.instanceId }} / .jobInstanceName: identify the exact instance, as shown in the console's Monitor Instance list.
• {{ outputs.evaluate.values.severity }}: OK, STUCK, STALE, NEEDS_ATTENTION or CRITICAL.

Pitfalls

• GetJobRun with instanceId omitted resolves the single most recent instance only; it is not a history query. A job run that failed three runs ago and has since succeeded will not be seen.
• allowFailure: true on get_run means an auth error, a typo in job_name, or a genuinely missing job all surface the same way, as CRITICAL with status: unknown. Check the task's own log on a CRITICAL result before assuming the job itself is the problem.
• max_staleness_hours should be set wider than the job's own schedule interval; setting it equal to or narrower than the schedule makes every run look STALE right before the next one fires.
• planTime and startTime can differ from each other by more than the job's schedule interval if the job was queued behind a busy cluster; STUCK and STALE can both fire on the same instance when that happens, and STUCK is checked first.
• DataArts requires the watched job to already have a schedule, HTTP trigger, or parent job configured in the console. A purely manual job with no trigger of any kind will never produce a run instance for get_run to find.

Quick start

1. Add the HUAWEI_AK, HUAWEI_SK and SLACK_WEBHOOK_URL secrets.
2. Set project_id, workspace_id (if applicable) and job_name to a real DataArts Factory job that has already run at least once.
3. Run the flow manually and confirm summary logs OK for a healthy job.
4. Lower max_staleness_hours below the job's actual last-run age to confirm alert fires with STALE.
5. Enable the schedule_hourly trigger once the thresholds match the job's real schedule.

How to extend

• Loop this flow over a list of job_name values with io.kestra.plugin.core.flow.Loop to watch an entire DataArts Factory workspace in one scheduled run.
• Add a remediation branch that calls io.kestra.plugin.huawei.dataarts.StartJobRun with a single scheduling period when severity is STALE, gated behind the same auto_remediate/dry_run pair used elsewhere in this repo's guard blueprints.
• Replace the Slack alert task with a Huawei Cloud SMN Publish task to page through Huawei's own notification service instead.

Links

• Huawei Cloud plugin documentation
• Huawei Cloud DataArts Studio API Reference
• Slack Incoming Webhooks