Post-Load Data Quality Scan with Soda

The cheapest place to catch bad data is the moment it lands, before a dashboard renders it or a model trains on it. This blueprint puts io.kestra.plugin.soda.Scan directly behind a load step: a DuckDB task stands in for your real ingestion, then Soda executes declarative SodaCL checks, row_count > 0 and missing_count(...) = 0, against the loaded table. Because a failed check fails the scan task itself, the flow has exactly two outcomes, a Slack verdict announcing a healthy load or a Slack alert routed through the errors block naming the execution to inspect.

How it works

1. load_orders (io.kestra.plugin.jdbc.duckdb.Queries) creates and refreshes a small orders table in the DuckDB file at warehouse_path. It exists so the blueprint runs anywhere; in production you replace it with your dlt, Airbyte, or JDBC load task.
2. scan_orders (io.kestra.plugin.soda.Scan) receives the warehouse connection through the configuration map, which must define a data source named kestra, and the SodaCL checks through the checks map, rendered to checks.yml inside the sodadata/soda-core container. The requirements list adds soda-core-duckdb for the DuckDB connection.
3. The task's failure semantics do the gating: a failed check sets the task state to FAILED and the execution jumps to the errors block, while warn-level results continue with a WARNING state. The parsed scan outcome is available under {{ outputs.scan_orders.result }}, including hasFailures, hasWarnings, hasErrors, the per-check checks list, and emitted metrics.
4. notify posts a verdict built from scalar outputs, the boolean flags and the check count, so the channel message is precise without dumping the full result payload.
5. A disabled-by-default Schedule trigger runs the scan every morning after the nightly loads should have landed.

What you get

• A quality gate with zero custom code: checks are declarative SodaCL YAML, versioned with the flow.
• Fail-loud semantics by construction, since a failing check fails the task and triggers the error alert.
• Scan results as structured task outputs, ready for downstream branching or reporting.
• Soda metrics emitted as Kestra metrics on every scan, giving you a quality time series in the execution history.

Who it's for

• Data engineers who want every load followed by an automatic quality verdict.
• Analytics teams tired of discovering NULL customer ids in a dashboard instead of a pipeline.
• Anyone adopting Soda who wants the smallest complete example of scan, verdict, and alert.

Why orchestrate this with Kestra

Soda answers whether the data is good; it does not decide when to scan, what happens on failure, or who gets told. Kestra supplies that half: the scan runs immediately after the load in the same flow, failure routing is explicit through the errors block, and every scan result is preserved as structured outputs in the execution history. The task also ships the Soda container image, so nothing is installed on workers.

Prerequisites

• A persistent path for the DuckDB file (the warehouse_path input) visible to the task containers. For production warehouses, swap the configuration map to Snowflake, BigQuery, or Postgres and add the matching soda-core-* package to requirements.
• A Slack incoming webhook for verdicts and alerts.

Secrets

• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the SLACK_WEBHOOK_URL secret to your Kestra namespace.
2. Execute the flow and confirm Slack reports 3 passing checks on the orders table.
3. Change a check to missing_count(customer_id) = -1 and rerun to watch the failure path exercise the errors block.
4. Replace load_orders with your real load task and enable the after_morning_load trigger.

How to extend

• Point the configuration map at Snowflake, BigQuery, or Postgres with credentials from {{ secret('...') }}, and add soda-core-snowflake, soda-core-bigquery, or soda-core-postgres to requirements.
• Add warn thresholds alongside fail thresholds, for example warn: when between 1 and 250, to get WARNING states without blocking the flow.
• Branch on {{ outputs.scan_orders.result.hasFailures }} with allowFailure: true on the scan, as shown in the Soda quality gate blueprint.
• Schedule a multi-table version of this scan, as shown in the Soda nightly warehouse audit blueprint.

Links

• Soda plugin documentation
• SodaCL reference