Run Tagged Dataform Models on Their Own Schedule

A whole-project dataform run is the wrong tool when the marketing team needs their models fresh at 06:00 and nobody wants to rebuild finance's fact tables to get there. Dataform solves the selection with tags: any SQLX action can declare tags: ["marketing"] in its config block, and dataform run --tags marketing executes just that slice with its dependencies. This blueprint parameterizes the tag as a Kestra input through io.kestra.plugin.dataform.cli.DataformCLI, so one flow serves every team's schedule, and each run reports to Slack.

How it works

1. The tag input (default marketing) selects the slice. Executing the flow manually with a different value runs a different team's models with zero YAML changes.
2. run_tagged_models (io.kestra.plugin.dataform.cli.DataformCLI) loads the project from namespace files, installs the pinned @dataform/core dependency in beforeCommands, and runs dataform run --tags {{ inputs.tag }} against BigQuery with credentials from inputFiles and GOOGLE_APPLICATION_CREDENTIALS.
3. Only actions carrying the tag execute, Dataform still resolves and orders their dependencies, so the slice is always internally consistent.
4. notify posts the finished run to Slack with the tag name, and the errors block posts a distinct alert on failure.
5. A disabled-by-default Schedule trigger runs the default tag every morning at 06:00. Duplicate the trigger, or the whole flow, per team and cadence.

What you get

• Per-team refresh schedules over one shared SQLX project, without splitting it into repositories.
• Cheaper runs, only the tagged actions and their dependencies consume BigQuery slots.
• Alerting scoped to the owning team, a failed marketing run never pages finance.
• One parameterized flow instead of a copy of the pipeline per tag.

Who it's for

• Analytics engineers running a shared Dataform project that serves teams with different freshness needs.
• Data platform teams replacing a monolithic nightly rebuild with staggered, tag-scoped refreshes.
• Anyone whose 06:00 dashboard deadline is hostage to unrelated models finishing first.

Why orchestrate this with Kestra

Dataform selects and runs the tagged slice, but the cadence, the credentials, the retry behavior, and the notification live outside the CLI. Kestra provides them around the unchanged command: inputs make the tag a first-class parameter, schedules give each slice its own cadence, every run is an execution with logs, and secrets never touch the project files.

Prerequisites

• A Dataform project stored as namespace files, with tags declared in the config blocks of the models to scope.
• A BigQuery service account with permission to run jobs and write to the target datasets.
• A Slack incoming webhook for run reports.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: the service account key JSON, delivered to the task as sa.json.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the GCP_SERVICE_ACCOUNT_JSON and SLACK_WEBHOOK_URL secrets to your Kestra namespace.
2. Upload your Dataform project to the namespace files and tag a few models with marketing.
3. Execute the flow and check that only the tagged models rebuilt in BigQuery.
4. Set disabled: false on the morning_marketing trigger for the daily 06:00 refresh.

How to extend

• Add more schedule triggers with different tags passed through trigger inputs, one cadence per team.
• Run the full project on a weekly controlled window instead, as shown in the full refresh weekend blueprint.
• Chain the tagged run behind upstream data landing, as shown in the incremental refresh blueprint.
• Gate the slice on assertions before dashboards read it, as shown in the assertions gate blueprint.

Links

• Dataform plugin documentation
• Dataform documentation