CI Compile Check for Dataform Pull Requests

A SQLX file with a broken ref, a typo in a config block, or an invalid dependency compiles nowhere, but without a gate it still merges and breaks the next production run. Dataform ships the check for this: dataform compile fails whenever any definition in the project is invalid, without running a single BigQuery job. This blueprint wires that check into a webhook-triggered Kestra flow through io.kestra.plugin.dataform.cli.DataformCLI, so a pull request webhook or a CI step invokes the gate and Discord receives exactly one pass or fail verdict.

How it works

1. The pr_webhook trigger (io.kestra.plugin.core.trigger.Webhook) starts the flow on demand, secured by a webhook key from a secret, so a Git provider webhook or a pipeline step can invoke the gate with one HTTP request.
2. compile_check (io.kestra.plugin.dataform.cli.DataformCLI) loads the project from namespace files, runs dataform install in beforeCommands, then runs dataform compile with || compile_ok=0 so a broken project records a flag instead of failing the flow before the verdict.
3. The result is emitted through Kestra's output protocol, making it available as {{ outputs.compile_check.vars.compile_ok }}.
4. verdict (io.kestra.plugin.core.flow.If) branches on the flag and posts exactly one Discord message, pass or fail.
5. The errors block posts a distinct alert if the gate itself crashes, so a silent gate never masquerades as a green one.

What you get

• Broken SQLX caught at pull request time, before it can break a scheduled production run.
• A compile check that costs no BigQuery slots, compilation never runs a warehouse job.
• One verdict message per invocation, whether the project compiles or not.
• A webhook interface any CI system or Git provider can call with one HTTP request.

Who it's for

• Analytics engineering teams who review SQLX changes through pull requests and want an automated first reviewer.
• Data platform teams centralizing CI checks in Kestra instead of duplicating them across repository pipelines.
• Anyone who has merged a Dataform change that compiled on one laptop and nowhere else.

Why orchestrate this with Kestra

Running dataform compile in scattered CI scripts means every repository re-implements the same step and the results vanish with the CI run. In Kestra, the gate is one flow with a stable webhook URL, every invocation is an execution with logs and structured outputs, and the same flow can grow richer steps, a dry run on success, a ticket on repeated failures, without touching any repository's pipeline definition.

Prerequisites

• A Dataform project stored as namespace files, kept in sync with the branch under review.
• A BigQuery service account key, compilation resolves the project configuration even though no jobs run.
• A Discord incoming webhook for verdicts and crash alerts.
• A webhook key of your choosing stored as a secret.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: the service account key JSON, delivered to the task as sa.json.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.
• WEBHOOK_KEY: shared key that authorizes calls to the webhook trigger.

Quick start

1. Add the GCP_SERVICE_ACCOUNT_JSON, DISCORD_WEBHOOK_URL, and WEBHOOK_KEY secrets to your Kestra namespace.
2. Upload your Dataform project to the namespace files.
3. Call the webhook URL from a curl command and check that Discord posts a verdict.
4. Point your Git provider's pull request webhook at the same URL.

How to extend

• Check out the exact commit referenced in the webhook body with io.kestra.plugin.git.Clone instead of relying on synced namespace files.
• Promote a green compile straight to a full run, as shown in the compile and run blueprint.
• Add assertion execution as a deeper second gate, as shown in the assertions gate blueprint.
• Post the verdict back to the pull request with an HTTP Request task against your Git provider's API.

Links

• Dataform plugin documentation
• Dataform documentation