Databricks Unity Catalog Table Naming Sentinel

A landing or staging schema with a naming convention nobody enforces drifts the moment a one-off script or a manual notebook run creates a table that does not follow it; Unity Catalog itself has no built-in naming policy check, it accepts whatever name a CREATE TABLE statement sends. This blueprint watches a governed schema continuously and flags the exact table the moment it appears, not during the next schema audit someone remembers to run.

This blueprint was created by Mohit Jeswani.

How it works

1. watch_new_tables (io.kestra.plugin.databricks.unitycatalog.table.Trigger) polls catalog_name.schema_name every 5 minutes and fires one execution per batch of newly created tables, tracking what it has already seen in the namespace KV Store so the same table is never reported twice.
2. evaluate (io.kestra.plugin.core.output.OutputValues) checks each new table's own name (after the last dot in its fully qualified name) against required_name_prefix with a jq filter.
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs that every new table follows the convention.
   • alert: posts the non-compliant fully qualified names to Slack.
4. summary logs the new-table count and severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails, for example an expired Databricks token.

What you get

• A naming-convention check that runs the moment a table appears, not on the next manual audit of the schema.
• The exact fully qualified table names in the Slack alert, so whoever responds can rename or remove the offending table directly.
• A read-only check with zero risk of creating, modifying, or deleting a real table.

Who it's for

• Data platform and governance teams running Unity Catalog who enforce a naming convention on a landing or staging schema and want violations caught the moment they land.
• Teams evaluating the Databricks plugin who want a worked example of unitycatalog.table.Trigger used as a standing naming-compliance sentinel, distinct from this repo's existing Databricks Unity Catalog blueprint, which audits grant permissions via the CLI rather than reacting to new table creation.

Why orchestrate this with Kestra

Unity Catalog reports its tables on request; it does not keep a schedule, compare a new table's name against a convention you declare, or notify a Slack channel when one breaks it. Kestra keeps required_name_prefix as a flow input, turns a naming violation into a Slack page naming the exact table, and keeps a full execution history of every table ever detected.

Prerequisites

• A Databricks workspace with Unity Catalog enabled and at least one governed schema.
• A Databricks personal access token with read access to that catalog and schema.
• A Slack incoming webhook URL.

This flow calls the Databricks Unity Catalog API directly over HTTPS; there is no official local Unity Catalog emulator, so watch_new_tables cannot be exercised against a local Docker container.

Secrets

• DATABRICKS_HOST / DATABRICKS_TOKEN: the workspace URL and personal access token used by watch_new_tables. In Kestra OSS, provide them as the base64-encoded environment variables SECRET_DATABRICKS_HOST and SECRET_DATABRICKS_TOKEN.
• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
required_name_prefix	stg_	Prefix every new table in the monitored schema must start with

Outputs

• {{ trigger.tables }} / .fullNames / .size: the new tables this poll detected, as the Trigger reports them (without column definitions; chain unitycatalog.table.Get for full schema).
• {{ outputs.evaluate.values.noncompliant_tables }} / .new_table_count: the violating table names, semicolon-joined, and how many tables were detected in total.
• {{ outputs.evaluate.values.severity }}: OK or UNAPPROVED_TABLE_NAME.

Pitfalls

• on: CREATE (the default) means the first evaluation after enabling this trigger reports every existing table in the schema as "new," since the KV-stored state starts empty; expect one large initial alert if the schema already has non-compliant tables.
• The naming check only looks at the table's own name segment, not its catalog or schema; a compliant table name created in the wrong schema entirely is not caught by this check.
• This trigger has no inputs: to override per execution; catalogName, schemaName, and interval are trigger-level configuration set directly on the trigger block.

Quick start

1. Add the DATABRICKS_HOST, DATABRICKS_TOKEN, and SLACK_WEBHOOK_URL secrets.
2. Set catalogName and schemaName on watch_new_tables to a real governed schema.
3. Enable the trigger; expect an initial alert if the schema already has non-compliant table names, since on: CREATE reports every existing table as new on first run.
4. Create a table that violates required_name_prefix to confirm a fresh alert fires.

How to extend

• Switch on to CREATE_OR_UPDATE to also flag a compliant table later renamed or recreated to break the convention.
• Chain io.kestra.plugin.databricks.unitycatalog.table.Get on each non-compliant table to include its owner and creation timestamp in the alert.
• Add a second naming rule (for example, requiring a department tag in the name) and run it alongside this one in the same evaluate step.

Links

• Databricks plugin: https://kestra.io/plugins/plugin-databricks
• Unity Catalog naming conventions reference: https://docs.databricks.com/en/data-governance/unity-catalog/best-practices.html
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks