Huawei DLI Data Quality Assertion Gate

DLI (Data Lake Insight) is Huawei's serverless Spark SQL engine over OBS and federated sources - the Huawei Cloud equivalent of Amazon Athena. Nothing in DLI itself turns a failed data quality rule into a notification: you either query the console by hand or you don't know until a downstream job breaks. This blueprint runs a single SQL assertion on a schedule, compares its result to a threshold, and alerts the moment the rule is violated.

This blueprint was created by Mohit Jeswani.

How it works

1. run_assertion (io.kestra.plugin.huawei.dli.Query) submits assertion_sql as a DLI SQL job on queue and fetches the single result row directly via fetchType: FETCH_ONE. Read-only, as long as assertion_sql is a SELECT.
2. evaluate (io.kestra.plugin.core.output.OutputValues) reads violation_count from the result row and compares it to max_violation_count. A DLI job status other than FINISHED is treated as CRITICAL, the same as a violation.
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs the passing count.
   • alert: posts the violation count and job status to Slack.
4. summary logs the count and severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails outright, for example an expired AK/SK.

What you get

• A data quality rule expressed as plain SQL, not a new DSL to learn, running against data already queryable from DLI.
• A single, reusable gate: point assertion_sql at a different rule and reuse the same flow for null-key checks, duplicate-row checks, referential integrity checks, or any other count-shaped assertion.
• A severity that distinguishes an actual data quality violation from the assertion query itself failing to run, so a broken check never silently reads as "compliant".

Who it's for

• Data platform teams who query OBS-backed tables through DLI and want a quality gate without standing up a separate Great Expectations or Soda deployment for one or two rules.
• Teams evaluating the Huawei Cloud plugin who want a worked example of dli.Query used for a read-only assertion, distinct from the plugin's own documented STORE/export-to-OBS use case.

Why orchestrate this with Kestra

DLI runs the SQL; it does not decide a cadence, compare the result to a threshold, or notify anyone. Kestra keeps assertion_sql and max_violation_count as flow inputs instead of a hardcoded script, turns the single-row result into a Slack page with severity attached, and keeps a full execution history of every check, compliant or not.

Prerequisites

• A Huawei Cloud account with DLI enabled, a database, and a dedicated DLI SQL queue (the shared default queue cannot fetch results via the preview API this blueprint relies on).
• An AK/SK pair (IAM access key and secret key) with DLI query permission.
• A table or data source already queryable from DLI to assert against.
• A Slack incoming webhook URL.

This flow calls the Huawei Cloud DLI API directly over HTTPS; there is no official local DLI emulator, so run_assertion cannot be exercised against a local Docker container. Validate assertion_sql directly in the DLI console's SQL editor first, then point this blueprint at it.

Secrets

• HUAWEI_AK: the IAM access key used by run_assertion. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_HUAWEI_AK.
• HUAWEI_SK: the IAM secret key paired with HUAWEI_AK. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_HUAWEI_SK.
• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
region	eu-west-101	Huawei Cloud region used to sign DLI requests
database	<your-dli-database>	DLI currentdb for the assertion query
queue	<your-dli-queue>	Dedicated DLI SQL queue, required for FETCH_ONE
assertion_sql	a null-check example	SELECT returning one row aliased violation_count
max_violation_count	0	violation_count above this triggers the alert

Outputs

• {{ outputs.run_assertion.row.violation_count }}: the raw value returned by the assertion.
• {{ outputs.run_assertion.status }}: FINISHED, FAILED, or CANCELLED.
• {{ outputs.evaluate.values.severity }}: OK, DATA_QUALITY_VIOLATION, or CRITICAL.

Pitfalls

• assertion_sql must alias its single output column violation_count exactly; this blueprint reads that column name directly rather than taking "whatever the first column is", so a typo in the alias surfaces as a null comparison, not a clear error.
• fetchType: FETCH_ONE is not supported on DLI's shared default queue - queue must name a dedicated DLI SQL queue, or the task fails fast with DLI's own documented limitation.
• This gate trusts assertion_sql to be read-only. DLI will execute DDL/DML just as readily as a SELECT; nothing here stops an assertion query that was accidentally written as an UPDATE.
• DLI's preview API caps results at 1000 rows, irrelevant here since FETCH_ONE only ever reads the first row, but relevant if you adapt this blueprint's run_assertion task to use fetchType: FETCH instead for a multi-row assertion.

Quick start

1. Add the HUAWEI_AK, HUAWEI_SK and SLACK_WEBHOOK_URL secrets.
2. Validate assertion_sql in the DLI console against a real table, confirming it returns one row with a violation_count column.
3. Set database and queue to match, then run the flow and confirm summary logs a count.
4. Lower max_violation_count below the real count to confirm alert fires.
5. Enable the hourly trigger once the threshold reflects your actual data quality bar.

How to extend

• Loop over a list of assertion_sql values with io.kestra.plugin.core.flow.Loop to run several data quality rules against the same database in one scheduled run.
• Swap the Slack alert task for a Huawei Cloud SMN Publish task to page through Huawei's own notification service instead.
• Change fetchType to STORE and read the exported OBS file in a downstream task when a violation needs the actual offending rows, not just the count.

Links

• Huawei Cloud plugin: https://kestra.io/plugins/plugin-huawei
• DLI SQL syntax reference: https://support.huaweicloud.com/intl/en-us/sqlreference-dli/dli_08_0001.html
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks