Databricks Lakebase Stale Pending Record Sentinel

A record left in pending because a downstream worker crashed, or an order stuck processing because a webhook never arrived, sits in Lakebase looking identical to a record that is one second into normal processing. Lakebase itself has no built-in alert for "this has been pending too long," it is a database, not a workflow engine. This blueprint runs the age check server-side and reports the exact records still stuck, not a guess.

This blueprint was created by Mohit Jeswani.

How it works

1. query_stale_records (io.kestra.plugin.databricks.lakebase.Query) mints a short-lived OAuth database credential and runs stale_query over the Postgres wire protocol, so the age and status filtering happens server-side in one SQL statement instead of this flow paging through every row to check client-side. Read-only; nothing in Lakebase is changed. allowFailure: true covers a connection or credential error without failing the flow outright.
2. evaluate (io.kestra.plugin.core.output.OutputValues) reads the matched count directly from the query result's own size field and collects the matched record ids with a jq filter.
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs that nothing currently matches the stale condition.
   • alert: posts the count and record ids to Slack.
4. summary logs the count and severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails, for example an expired OAuth credential.

What you get

• A count that is the real query result, not a client-side guess computed after paging through the whole table.
• The exact stale record ids in the Slack alert, so whoever responds can jump straight to the record instead of re-running the query themselves.
• A read-only check with zero risk of updating or deleting a real record.

Who it's for

• Data platform and backend teams running Databricks Lakebase who need a status-stuck record caught on a schedule, not discovered when a customer asks why their order never moved past pending.
• Teams evaluating the Databricks plugin who want a worked example of lakebase.Query used as a standing governance check, distinct from this repo's existing Databricks blueprints, which cover Unity Catalog grants and tables rather than Lakebase's own OLTP tables.

Why orchestrate this with Kestra

Lakebase answers one query at a time; it does not keep a schedule, compare the match count against a tolerance you declare, or notify a Slack channel when stuck records pile up. Kestra keeps stale_query and max_stale_count as flow inputs, turns the query into a Slack page naming the real record ids, and keeps a full execution history of every run.

Prerequisites

• A Databricks workspace with a Lakebase instance and a database containing a table with a status and an updated-at column.
• A Databricks service principal (clientId/clientSecret) with access to mint a Lakebase database credential for that instance.
• A Slack incoming webhook URL.

This flow connects to Lakebase over the Postgres wire protocol directly; there is no official local Lakebase emulator, so query_stale_records cannot be exercised against a local Docker container, though any standard postgres image can be used to prototype the SQL itself before pointing it at a real Lakebase instance.

Secrets

• DATABRICKS_HOST / DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET / LAKEBASE_ENDPOINT_NAME: the workspace host, service principal credentials, and Lakebase endpoint name used by query_stale_records. In Kestra OSS, provide them as the base64-encoded environment variables SECRET_DATABRICKS_HOST, SECRET_DATABRICKS_CLIENT_ID, SECRET_DATABRICKS_CLIENT_SECRET, and SECRET_LAKEBASE_ENDPOINT_NAME.
• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
lakebase_database	<your-lakebase-database>	Lakebase database to query
stale_query	SELECT id, status, updated_at FROM orders WHERE status = 'pending' AND updated_at < now() - interval '1 hour'	SQL selecting the stale records
max_stale_count	1	Matched-record count that triggers the Slack alert

Outputs

• {{ outputs.query_stale_records.rows }}: every matched record, as returned by stale_query.
• {{ outputs.evaluate.values.stale_count }} / .stale_ids: how many records matched and their ids, semicolon-joined.
• {{ outputs.evaluate.values.severity }}: OK, STALE_RECORDS_FOUND, or CRITICAL.

Pitfalls

• stale_query must return a column literally named id; the jq filter that collects record ids for the Slack alert looks for that exact key, a differently-named primary key column needs the query aliased as id.
• stale_query is the entire governance policy; a typo in the SQL surfaces via allowFailure, not a silently empty result.
• fetchType: FETCH keeps every matched row in memory for the jq filter; a query that can legitimately match a very large number of rows should lower fetchType to STORE and alert with the storage URI instead.

Quick start

1. Add the four Databricks/Lakebase secrets and SLACK_WEBHOOK_URL.
2. Set lakebase_database and stale_query to match a real table and status column.
3. Run the flow manually and confirm summary logs a count.
4. Insert or update a row to match stale_query to confirm alert fires with its id.
5. Enable the hourly trigger once stale_query reflects your real SLA policy.

How to extend

• Loop over several stale_query values with io.kestra.plugin.core.flow.Loop to enforce a different staleness policy per table in one scheduled run.
• Use afterSQL on a follow-up lakebase.Query task to mark matched records as needs_review in the same transaction, gated behind an explicit confirm_flag input.
• Chain io.kestra.plugin.databricks.unitycatalog.table.Get if the same data is also mirrored into Unity Catalog, to cross-check both representations agree.

Links

• Databricks plugin: https://kestra.io/plugins/plugin-databricks
• Databricks Lakebase overview: https://docs.databricks.com/aws/en/oltp/
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks