Compile and Run Dataform Transformations on BigQuery

Dataform turns SQL transformations into a dependency-aware DAG of SQLX files, but the CLI still needs somewhere to run on schedule, with credentials, logs, and alerting. This blueprint executes the whole lifecycle through io.kestra.plugin.dataform.cli.DataformCLI: the project arrives from Kestra namespace files, dataform compile validates every SQLX definition before anything touches the warehouse, dataform run executes the compiled DAG against BigQuery, and Slack receives the outcome with the execution ID. Every transformation run becomes a versioned execution with full logs.

How it works

1. compile_and_run (io.kestra.plugin.dataform.cli.DataformCLI) loads the Dataform project from namespace files with namespaceFiles: enabled, so the SQLX definitions live in the Kestra editor or sync from Git.
2. inputFiles delivers the BigQuery service account key from {{ secret('GCP_SERVICE_ACCOUNT_JSON') }} as sa.json, and env points GOOGLE_APPLICATION_CREDENTIALS at it. Keep the project's .df-credentials.json (created by dataform init-creds) in the namespace files alongside the SQLX sources.
3. beforeCommands runs dataform install to fetch the project's pinned @dataform/core dependency before any compilation.
4. commands runs dataform compile first, so a broken ref, a missing dependency, or invalid SQLX stops the execution before any BigQuery job starts, then dataform run executes the full DAG in dependency order.
5. notify posts the outcome to Slack, and the errors block posts a distinct alert when compile or run fails.
6. A disabled-by-default Schedule trigger rebuilds the models daily at 05:00.

What you get

• Every transformation run as an auditable execution with logs, timing, and the exact project state that ran.
• Compile-before-run by default, so definition errors never surface as half-built warehouse state.
• Credentials from Kestra's secret backend instead of a keyfile sitting in a repository or on a VM.
• A pattern that scales from one model to a full project without changing the flow.

Who it's for

• Analytics engineers moving Dataform runs off cron boxes and laptops into versioned, logged executions.
• Data teams on BigQuery who want SQLX transformations orchestrated next to ingestion and reverse ETL, not in a separate scheduler.
• Anyone adopting Dataform who wants a working compile and run pipeline in one copy-paste.

Why orchestrate this with Kestra

Dataform compiles and runs the SQL DAG, but it does not schedule itself, keep an execution history, hold credentials safely, or tell anyone when a model breaks. Kestra wraps the unchanged dataform commands with all of that: the project files live in namespace files or Git, every command's logs land in the execution view, secrets stay in Kestra's secret backend, and downstream flows can chain on completion.

Prerequisites

• A Dataform project (SQLX sources, dataform.json or workflow_settings.yaml, and a .df-credentials.json created by dataform init-creds) stored as namespace files.
• A BigQuery service account with permission to run jobs and write to the target datasets.
• A Slack incoming webhook for run reports.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: the service account key JSON, delivered to the task as sa.json.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the GCP_SERVICE_ACCOUNT_JSON and SLACK_WEBHOOK_URL secrets to your Kestra namespace.
2. Upload your Dataform project to the namespace files, including .df-credentials.json.
3. Execute the flow and check that Slack reports the finished run and the models appear in BigQuery.
4. Set disabled: false on the daily trigger to run every morning.

How to extend

• Run only a subset of models with dataform run --tags, as shown in the tag-scoped run blueprint.
• Gate downstream consumers on data quality by capturing assertion results, as shown in the assertions gate blueprint.
• Trigger the run the moment upstream data lands instead of on a timer, as shown in the incremental refresh blueprint.
• Sync the project from a Git repository into namespace files so the same SQLX drives development and production.

Links

• Dataform plugin documentation
• Dataform documentation