Data Freshness Checks with Soda

A table full of last Tuesday's rows passes every row_count and missing_count check while being wrong for every decision made on it. This blueprint treats staleness as a first-class quality failure: io.kestra.plugin.soda.Scan runs a SodaCL freshness check on the newest created_at value with two thresholds. Past 6 hours the check warns and the task ends in a WARNING state, a visible early signal that data is aging; past 24 hours the check fails, the task fails, and the errors block tells Slack to stop trusting the table.

How it works

1. load_events (io.kestra.plugin.jdbc.duckdb.Queries) refreshes a small events table with current timestamps in the DuckDB file at warehouse_path. It exists so the blueprint runs anywhere and passes on first run; in production your streaming sink or batch load keeps the table fresh.
2. freshness_scan (io.kestra.plugin.soda.Scan) receives the connection through the configuration map, which must define a data source named kestra, and the tiered check through the checks map: freshness(created_at) with warn: when > 6h and fail: when > 24h. The requirements list adds soda-core-duckdb.
3. The tiers map directly onto Kestra states: a warn-level result continues the flow with a WARNING state, while a fail-level result sets the task to FAILED and jumps to the errors block. Staleness escalates the same way any other quality failure does.
4. notify posts the verdict with the scalar hasWarnings and hasFailures flags from {{ outputs.freshness_scan.result }}, so the channel sees the warn tier while there is still time to fix the upstream load.
5. A disabled-by-default Schedule trigger probes every 6 hours, matching the warn threshold so aging data is caught within one warn window.

What you get

• Staleness handled with the same machinery as any quality failure, not as a separate monitoring stack.
• A two-tier escalation, WARNING while the table ages and a hard failure with an alert once it breaches the contract.
• A freshness contract declared in three lines of SodaCL, versioned with the flow.
• Executions that record measured freshness over time, since Soda emits its metrics as Kestra metrics on every scan.

Who it's for

• Data engineers whose consumers assume tables are current unless told otherwise.
• Analytics teams that have shipped a report on data that stopped loading two days earlier.
• Anyone monitoring freshness with hand-written SQL age queries who wants them declarative.

Why orchestrate this with Kestra

A freshness check is a measurement plus a policy: how often to probe, what a warning means, who hears about a breach. Kestra supplies the policy half. The Schedule trigger sets the probe cadence, warn and fail tiers map onto WARNING and FAILED execution states visible in every run, the errors block makes the breach loud, and the execution history becomes a freshness log for the table. The task ships the Soda container image, so nothing is installed on workers.

Prerequisites

• A persistent path for the DuckDB file (the warehouse_path input) visible to the task containers. For production warehouses, swap the configuration map to Snowflake, BigQuery, or Postgres and add the matching soda-core-* package to requirements.
• A timestamp column that reflects load or event time, such as created_at.
• A Slack incoming webhook for verdicts and alerts.

Secrets

• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the SLACK_WEBHOOK_URL secret to your Kestra namespace.
2. Execute the flow and confirm Slack reports the table within the freshness contract.
3. Change the fail threshold to when > 0m and rerun to watch the staleness path exercise the errors block.
4. Point the check at your real table and timestamp column, set thresholds to your load cadence, and enable the freshness_probe trigger.

How to extend

• Point the configuration map at Snowflake, BigQuery, or Postgres with credentials from {{ secret('...') }}, and add soda-core-snowflake, soda-core-bigquery, or soda-core-postgres to requirements.
• Add freshness checks for more tables as additional checks for <table> blocks with per-table thresholds.
• Combine freshness with row_count and missing_count checks in the same scan, so one task answers both whether data arrived and whether it is any good.
• Escalate a freshness breach to an incident, as shown in the Soda incident escalation blueprint.

Links

• Soda plugin documentation
• SodaCL freshness checks