Unity Catalog Grant Compliance Gate with Kestra

id: databricks-unity-catalog-grant-compliance-gate
namespace: company.team

description: |
  Audit who holds high-risk Unity Catalog privileges on a securable object, alert Slack when a
  principal outside the allow list holds one, and optionally revoke the violating privileges
  with the Databricks CLI's own grants update command behind a dry_run gate, then re-probe to
  verify the revoke actually worked.

inputs:
  - id: securable_type
    type: SELECT
    displayName: Unity Catalog securable type
    description: Type of the Unity Catalog object to audit. Maps to the Get and
      Update tasks' securableType field.
    values:
      - "CATALOG"
      - "SCHEMA"
      - "TABLE"
    defaults: "SCHEMA"

  - id: full_name
    type: STRING
    displayName: Securable full name
    description: Full name of the securable, catalog, catalog.schema, or
      catalog.schema.table depending on securable_type.
    defaults: "main.landing_zone"

  - id: allowed_principals
    type: JSON
    displayName: Allow-listed principals
    description: Users, groups or service principals permitted to hold a
      high_risk_privileges privilege on this securable. Anyone else holding one
      is a violation.
    defaults: '["data-platform-admins"]'

  - id: high_risk_privileges
    type: JSON
    displayName: High-risk privileges to audit
    description: Privileges that only an allow-listed principal may hold on this
      securable, for example ALL_PRIVILEGES or MODIFY.
    defaults: '["ALL_PRIVILEGES", "MODIFY"]'

  - id: auto_remediate
    type: BOOL
    displayName: Auto remediate
    description: When true, together with dry_run false, revoke every violating
      privilege found via the Databricks CLI's grants update command.
    defaults: false

  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: Must be false, together with auto_remediate true, for the revoke to
      actually run.
    defaults: true

tasks:
  - id: probe_grants
    type: io.kestra.plugin.databricks.cli.DatabricksCLI
    description: Read-only. Runs the Databricks CLI's own grants get command and
      writes its JSON response to internal storage for the audit step to read.
    allowFailure: true
    databricksHost: "{{ secret('DATABRICKS_HOST') }}"
    databricksToken: "{{ secret('DATABRICKS_TOKEN') }}"
    commands:
      - databricks grants get {{ inputs.securable_type }} {{ inputs.full_name }}
        --output json > grants.json
    outputFiles:
      - grants.json

  - id: audit_grants
    type: io.kestra.plugin.core.output.OutputValues
    description: Flags every principal outside the allow list that holds a high-risk
      privilege. Empty violations when the probe itself did not return. Field
      names (privilege_assignments) match the Databricks Grants REST API
      response verbatim, since the CLI's --output json passes the API response
      straight through.
    values:
      violations: >-
        {{ (outputs.probe_grants is not defined) ? '[]' : ({'assignments':
        (fromJson(read(outputs.probe_grants.outputFiles['grants.json'])).privilege_assignments
        ?? []), 'allowed': inputs.allowed_principals, 'risky':
        inputs.high_risk_privileges} | jq('.allowed as $a | .risky as $r |
        [(.assignments // [])[] | . as $x | ($x.privileges // [] | map(select(.
        as $p | $r | index($p) != null))) as $hit | select(($hit | length > 0)
        and (($x.principal as $pr | $a | index($pr)) == null)) | {principal:
        $x.principal, matched_privileges: $hit}]') | first | toJson) }}

  - id: log_summary
    type: io.kestra.plugin.core.log.Log
    message: "Unity Catalog grant audit on {{ inputs.securable_type }} {{
      inputs.full_name }}: {{ fromJson(outputs.audit_grants.values.violations) |
      length }} violation(s)."

  - id: route_compliance
    type: io.kestra.plugin.core.flow.Switch
    description: Branch A logs a compliant securable. Branch B alerts Slack when a
      non-allow-listed principal holds a high-risk privilege, or when the probe
      itself failed.
    value: "{{ outputs.probe_grants is not defined ? 'PROBE_FAILED' :
      (fromJson(outputs.audit_grants.values.violations) | length > 0 ?
      'VIOLATION' : 'COMPLIANT') }}"
    cases:
      "PROBE_FAILED":
        - id: alert_probe_failed
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {"text": "Unity Catalog grant probe for " ~ inputs.securable_type ~ " " ~ inputs.full_name ~ " did not return a result. Check DATABRICKS_HOST, DATABRICKS_TOKEN, and that the securable exists."} | toJson }}
      "VIOLATION":
        - id: alert_violation
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {"text": "Unity Catalog compliance VIOLATION on " ~ inputs.securable_type ~ " " ~ inputs.full_name ~ ". " ~ (fromJson(outputs.audit_grants.values.violations) | length) ~ " principal(s) outside the allow list hold a high-risk privilege: " ~ (fromJson(outputs.audit_grants.values.violations) | jq('map("\(.principal): \(.matched_privileges | join(\", \"))") | join("; ")') | first) ~ "." ~ (inputs.auto_remediate and not inputs.dry_run ? " Revoking now." : " No change applied (auto_remediate=" ~ inputs.auto_remediate ~ ", dry_run=" ~ inputs.dry_run ~ ").")} | toJson }}

        - id: remediate_gate
          type: io.kestra.plugin.core.flow.If
          condition: "{{ inputs.auto_remediate and not inputs.dry_run }}"
          then:
            - id: revoke_violations
              type: io.kestra.plugin.core.flow.Loop
              concurrencyLimit: 1
              values: "{{ outputs.audit_grants.values.violations }}"
              tasks:
                - id: revoke_grant
                  type: io.kestra.plugin.databricks.cli.DatabricksCLI
                  description: Calls the Databricks CLI's own grants update command with a Grants
                    REST API-shaped --json body
                    (changes[].principal/add/remove), revoking only the
                    privileges that caused this violation for this principal.
                  databricksHost: "{{ secret('DATABRICKS_HOST') }}"
                  databricksToken: "{{ secret('DATABRICKS_TOKEN') }}"
                  commands:
                    - |
                      databricks grants update {{ inputs.securable_type }} {{ inputs.full_name }} --output json --json '{{ {'changes': [{'principal': (item.value | jq('.principal') | first), 'remove': (item.value | jq('.matched_privileges') | first)}]} | toJson }}'

            - id: reprobe_grants
              type: io.kestra.plugin.databricks.cli.DatabricksCLI
              description: Read-only re-check after the revoke, same command as probe_grants.
              allowFailure: true
              databricksHost: "{{ secret('DATABRICKS_HOST') }}"
              databricksToken: "{{ secret('DATABRICKS_TOKEN') }}"
              commands:
                - databricks grants get {{ inputs.securable_type }} {{
                  inputs.full_name }} --output json > grants.json
              outputFiles:
                - grants.json

            - id: reaudit_grants
              type: io.kestra.plugin.core.output.OutputValues
              values:
                violations: >-
                  {{ (outputs.reprobe_grants is not defined) ? '[]' :
                  ({'assignments':
                  (fromJson(read(outputs.reprobe_grants.outputFiles['grants.json'])).privilege_assignments
                  ?? []), 'allowed': inputs.allowed_principals, 'risky':
                  inputs.high_risk_privileges} | jq('.allowed as $a | .risky as
                  $r | [(.assignments // [])[] | . as $x | ($x.privileges // []
                  | map(select(. as $p | $r | index($p) != null))) as $hit |
                  select(($hit | length > 0) and (($x.principal as $pr | $a |
                  index($pr)) == null)) | {principal: $x.principal,
                  matched_privileges: $hit}]') | first | toJson) }}

            - id: verify_remediation
              type: io.kestra.plugin.core.flow.If
              description: Fails the flow when a revoke ran but a violation still remains, so
                a broken revoke is never silently reported as fixed.
              condition: "{{ fromJson(outputs.reaudit_grants.values.violations) | length > 0
                }}"
              then:
                - id: remediation_failed
                  type: io.kestra.plugin.core.execution.Fail
                  errorMessage: "grants update ran, but a violation still remains on this
                    securable."
              else:
                - id: remediated
                  type: io.kestra.plugin.core.log.Log
                  message: "Revoked and verified. 0 violations remain on {{ inputs.securable_type
                    }} {{ inputs.full_name }}."
      "COMPLIANT":
        - id: log_compliant
          type: io.kestra.plugin.core.log.Log
          message: "No violations. Every principal holding a high-risk privilege on {{
            inputs.securable_type }} {{ inputs.full_name }} is on the allow
            list."

  - id: log_audit_summary
    type: io.kestra.plugin.core.log.Log
    description: Final audit log confirming the gate ran, regardless of the branch
      taken above.
    message: "Unity Catalog grant compliance gate completed. securable={{
      inputs.securable_type }} {{ inputs.full_name }} auto_remediate={{
      inputs.auto_remediate }} dry_run={{ inputs.dry_run }}"

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {"text": "Unity Catalog grant compliance gate flow failed (execution " ~ execution.id ~ "). Check the execution logs."} | toJson }}

triggers:
  - id: schedule_every_6_hours
    type: io.kestra.plugin.core.trigger.Schedule
    description: Shipped disabled, and always in safe mode by default.
    cron: "0 */6 * * *"
    disabled: true
    inputs:
      securable_type: "SCHEMA"
      full_name: "main.landing_zone"
      allowed_principals: '["data-platform-admins"]'
      high_risk_privileges: '["ALL_PRIVILEGES", "MODIFY"]'
      auto_remediate: false
      dry_run: true

Unity Catalog access is granted one GRANT statement at a time, often months apart, by different people, with no single place that shows who currently holds ALL_PRIVILEGES or MODIFY on a given catalog, schema or table. A grant given for a one-off migration and never revoked is exactly how an unrelated team ends up able to drop a production schema. This blueprint turns a Unity Catalog permission check into a Day 2 governance gate: it reads the real privilege assignments on a securable, flags any principal outside your allow list holding a high-risk privilege, alerts Slack, and can revoke the excess privilege itself, then re-checks to prove the revoke actually took.

The plugin-databricks tasks this blueprint runs on top of (io.kestra.plugin.databricks.cli.DatabricksCLI) wrap the official Databricks CLI's own grants get and grants update commands, rather than a dedicated Unity Catalog grant task, since the plugin does not ship one as a released, published task today.

How it works

probe_grants (io.kestra.plugin.databricks.cli.DatabricksCLI) runs databricks grants get <securable_type> <full_name> --output json, redirected to grants.json and declared via outputFiles so downstream tasks can read it, with allowFailure: true so an auth or not-found error does not stop the flow before it can be classified and alerted on. The command's JSON is the Unity Catalog Grants REST API response verbatim: {"privilege_assignments": [{"principal", "privileges"}]}.
audit_grants (io.kestra.plugin.core.output.OutputValues) reads that file with read() and fromJson(), then compares each {principal, privileges} entry against allowed_principals and high_risk_privileges with a jq filter, returning the list of violations: principals outside the allow list holding at least one high-risk privilege. If probe_grants produced no output at all, this returns an empty list rather than failing, so the flow can still classify the run as PROBE_FAILED downstream instead of crashing.
log_summary (io.kestra.plugin.core.log.Log) logs the violation count.
route_compliance (io.kestra.plugin.core.flow.Switch) branches on outputs.probe_grants is not defined (classified PROBE_FAILED) or the violation count (classified VIOLATION or COMPLIANT):
- COMPLIANT: log_compliant logs it, nothing else happens.
- PROBE_FAILED: a SlackIncomingWebhook task alerts that the probe itself did not return.
- VIOLATION: a SlackIncomingWebhook task alerts with the exact principal and privileges found, built only from audit_grants' own output. remediate_gate (io.kestra.plugin.core.flow.If) then runs only when auto_remediate is true and dry_run is false:
  - revoke_violations (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) calls databricks grants update <securable_type> <full_name> --json '...' once per violation, with a body shaped exactly like the Grants REST API's update endpoint ({"changes": [{"principal", "remove": [...]}]}), revoking only the privileges that caused the violation.
  - reprobe_grants runs the same grants get command again, reaudit_grants re-runs the same jq filter, and verify_remediation (io.kestra.plugin.core.flow.If) fails the flow with io.kestra.plugin.core.execution.Fail if a violation still remains, so a partially-applied revoke is never silently reported as fixed.
log_audit_summary always runs last and logs the final state of the gate.
The errors block posts to Slack if the flow itself fails outside the gate logic above.

What you get

A point-in-time answer to "who can do what" on a Unity Catalog securable, instead of trusting that the last GRANT was the last one needed.
A Slack alert that names the exact principal and the exact privileges that triggered it, never an invented number.
A safe-by-default revoke path: dry_run: true ships enabled, so nothing is ever changed until you explicitly turn both auto_remediate on and dry_run off.
A verified remediation: the flow re-reads the grants after revoking and fails loudly if the violation is still there.

Who it's for

Data platform and data governance teams who own Unity Catalog access control and need to prove, on a schedule, that only approved principals hold high-risk privileges.
Security and compliance teams who need an audit trail of access reviews on sensitive catalogs or schemas, beyond a point-in-time manual review.
Teams evaluating the Databricks plugin who want a working example of cli.DatabricksCLI driving the Databricks CLI's own grants get and grants update commands, captured back into Kestra via outputFiles and read()/fromJson().

Why orchestrate this with Kestra

Unity Catalog's own GRANT/REVOKE statements have no concept of a schedule, an allow list, or an alert, they just apply the change you ask for. Kestra adds the recurring audit, the allow-list comparison, the Slack alert carrying the exact violation, the gated revoke behind auto_remediate and dry_run, and the verification step that proves the revoke worked, all as one declarative YAML file you can version and review like code.

Prerequisites

A Databricks workspace with Unity Catalog enabled and a securable (catalog, schema or table) to audit. Databricks is a managed service with no self-hostable open-source Docker image, there is no local container to start for Databricks itself. DatabricksCLI pulls the official ghcr.io/databricks/cli:latest image automatically, no manual setup needed for the CLI itself.
A personal access token (or OAuth M2M service principal) with MANAGE on the securable, so the flow can both read (grants get) and, if auto_remediate is enabled, revoke (grants update) privileges.
A Slack incoming webhook for alerts.

Secrets

DATABRICKS_HOST: your workspace URL, for example https://<workspace>.cloud.databricks.com.
DATABRICKS_TOKEN: a personal access token with MANAGE on the securable.
SLACK_WEBHOOK_URL: Slack incoming webhook URL used by the alert tasks and the errors block.

In Kestra OSS, set all three as base64-encoded environment variables: SECRET_DATABRICKS_HOST, SECRET_DATABRICKS_TOKEN, SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Type	Default	Purpose
`securable_type`	SELECT	`SCHEMA`	Unity Catalog securable type (`CATALOG`, `SCHEMA`, `TABLE`)
`full_name`	STRING	`main.landing_zone`	Full name of the securable object
`allowed_principals`	JSON	`["data-platform-admins"]`	JSON array of approved principal names
`high_risk_privileges`	JSON	`["ALL_PRIVILEGES", "MODIFY"]`	JSON array of high-risk privileges to audit
`auto_remediate`	BOOL	`false`	When true with `dry_run: false`, revokes violating grants
`dry_run`	BOOL	`true`	Must be false to apply grant revocations

Quick start

Create a Databricks personal access token with MANAGE on the securable you want to audit.
Add the DATABRICKS_HOST, DATABRICKS_TOKEN, and SLACK_WEBHOOK_URL secrets.
Run this flow manually with the defaults and read log_audit_summary for the result.
If it reports a VIOLATION you expect, set auto_remediate: true and dry_run: false and run it again to revoke and verify.
Enable the schedule_every_6_hours trigger, which always ships in safe mode.

How to extend

Point full_name at a CATALOG and widen allowed_principals to run the same audit one level up the Unity Catalog hierarchy.
Add a second high_risk_privileges tier (for example SELECT on a PII-tagged schema) with its own allow list, and run a second probe_grants/audit_grants pair against it.
Read principal from an identity provider's group export instead of a static allowed_principals list, to catch access that is technically allow-listed but the group membership itself has drifted.
Chain a downstream task after COMPLIANT that writes the clean audit result to a Kestra KV entry, to track how long the securable has stayed compliant.

Pitfalls

databricks grants get only returns privileges granted directly on the securable, inherited privileges from a parent catalog or schema are not included, a principal can still have effective access through inheritance even when this audit reports zero direct grants.
The token used for grants update needs MANAGE on the securable, a token scoped only to read access will pass probe_grants but fail the revoke with a permission error.
securable_type is case-insensitive on the Databricks API side but must be one of CATALOG, SCHEMA, TABLE, VOLUME, EXTERNAL_LOCATION or STORAGE_CREDENTIAL, anything else is rejected before the API call is made.
revoke_grant's --json argument is built with a Pebble map literal and shell single-quoted; a principal name or privilege string containing a literal single quote would break the shell quoting. Unity Catalog principal names (emails, group names, service principal IDs) and privilege names are not expected to contain one in practice.
Revoking a privilege a principal needs for a job that runs outside this flow's schedule breaks that job at its next run, review allowed_principals against every consumer of the securable before enabling auto_remediate.

Databricks Unity Catalog Grant Compliance Gate

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Inputs

Quick start

How to extend

Pitfalls

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Databricks Unity Catalog Grant Compliance Gate

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Inputs

Quick start

How to extend

Pitfalls

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Data Blueprints

AES-256 Argon2id Cryptographic Landing and Compliance Gate

AI Document Data Extraction: Parse PDFs and Web Pages into Structured JSON with OpenAI

Actian Vector Table Fragmentation and Storage Sentinel

AI-Powered Inventory Demand Forecasting, Auto-Generate Purchase Orders with OpenAI

Aerospike Stop-Writes Headroom Sentinel

AI Data Quality Anomaly Advisor and SQL Remediation Generator

More Infrastructure Blueprints

ADLS World-Writable ACL Sentinel

Apache ActiveMQ Artemis Dead Letter Address (DLA) Message Accumulation Sentinel

Aerospike Stop-Writes Headroom Sentinel

AI Incident Post-Mortem and Root Cause Generator

Automatically Review GitHub Pull Requests with OpenAI GPT-4o and Post Comments

Environmental Disaster Early Warning System

New to Kestra?