dbt Orchestration: Running dbt as Part of Production Pipelines

TL;DR — dbt orchestration is the layer that decides when dbt runs and what surrounds it: it triggers builds after ingestion, runs only the models that changed, retries failures, alerts the team, and connects dbt to the systems upstream and downstream. dbt defines the transformations; the orchestrator makes them a reliable production pipeline.

dbt has revolutionized how data teams transform data in their warehouses, bringing software engineering best practices to analytics. Yet dbt only reaches production when it is paired with an orchestrator. While dbt excels at defining transformations, it relies on external systems to manage its lifecycle: from triggering runs based on events to handling complex dependencies, ensuring state-aware execution, and integrating with the broader data and infrastructure stack. This article explores why dedicated orchestration is essential for dbt projects and how to build resilient, production-grade dbt pipelines.

How dbt orchestration works

At its core, dbt orchestration operates on a principle of separation of concerns. dbt handles the “what” of data transformation: defining models, tests, and dependencies within the data warehouse. An orchestrator handles the “when” and “how”: coordinating the execution of these transformations within a larger workflow.

The orchestrator’s job is to trigger dbt CLI commands (like dbt build or dbt test) at the right time and in the right context. This involves:

• Scheduling: Running dbt jobs on a fixed schedule (e.g., daily, hourly).
• Event-driven triggers: Initiating dbt runs based on external events, such as the arrival of new data from an ingestion tool.
• Dependency management: Ensuring that upstream tasks (like data loading) are complete before starting a dbt run, and that downstream tasks (like updating a BI dashboard) are triggered upon its successful completion.
• Parameterization: Passing dynamic parameters to dbt runs, such as dates or environment-specific configurations.

This coordination turns standalone dbt models into reliable components of a fully automated data pipeline. A well-defined orchestration flow provides the control and visibility necessary for production environments.

Why dbt projects need an orchestrator

While dbt Cloud offers a native scheduler, production data platforms often require more sophisticated coordination that extends beyond the capabilities of a simple job scheduler.

• Beyond basic scheduling: Cron-based scheduling is a starting point, but real-world pipelines need to react to events. A dbt run should start when new data lands from Airbyte, not at a fixed time, to ensure freshness and efficiency.
• Dependency management: A dbt model is rarely an island. The orchestrator manages the entire chain, from data ingestion and data quality checks to running dbt models and then feeding the results into reverse ETL tools or machine learning models.
• Error handling & recovery: Production pipelines fail. An orchestrator provides recovery mechanisms like automated retries with exponential backoff, custom alerting to Slack or PagerDuty, and the ability to define specific failure branches to handle issues gracefully without manual intervention.
• State-aware execution: Rebuilding all dbt models on every run is inefficient and costly. State-aware orchestration runs only the models impacted by changes in data or code, significantly reducing computation time and warehouse costs.
• CI/CD integration: A mature CI/CD orchestration process for dbt involves more than just running tests. It includes automatically deploying projects, managing different environments (dev, staging, prod), and promoting artifacts in a governed way.
• Resource management: Orchestrators can run dbt jobs in isolated environments, such as Docker containers. This ensures that each run has the exact dependencies it needs, preventing conflicts and making the pipeline more portable and reproducible.

Orchestrate dbt with Kestra: a state-aware pipeline example

Kestra provides a declarative, YAML-based approach to orchestrating dbt workflows. Instead of writing Python DAGs, you define your entire pipeline, including Git operations, dbt commands, and notifications, in a single version-controlled configuration file.

The following example demonstrates a daily, state-aware dbt pipeline. It automatically clones the dbt project from a Git repository, runs only the modified models, and sends a detailed Slack alert if any step fails.

id: dbt_state_aware_daily_run
namespace: company.team
description: Runs dbt daily on the models changed since the last run, and alerts Slack on failure.

inputs:
  - id: dbt_git_repo
    type: STRING
    defaults: "https://github.com/your-org/your-dbt-project.git"
  - id: dbt_git_branch
    type: STRING
    defaults: main
  - id: dbt_profile
    type: STRING
    defaults: your_warehouse_profile

tasks:
  - id: dbt
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone_dbt_project
        type: io.kestra.plugin.git.Clone
        url: "{{ inputs.dbt_git_repo }}"
        branch: "{{ inputs.dbt_git_branch }}"

      - id: dbt_build_state_aware
        type: io.kestra.plugin.dbt.cli.DbtCLI
        containerImage: ghcr.io/dbt-labs/dbt-postgres:1.8.0
        loadManifest:
          key: manifest.json
          namespace: "{{ flow.namespace }}"
        storeManifest:
          key: manifest.json
          namespace: "{{ flow.namespace }}"
        commands:
          - dbt deps
          - dbt build --select state:modified+ --defer --state ./target
        profiles: |
          {{ inputs.dbt_profile }}:
            target: prod
            outputs:
              prod:
                type: postgres
                host: "{{ secret('DBT_HOST') }}"
                port: 5432
                user: "{{ secret('DBT_USER') }}"
                password: "{{ secret('DBT_PASSWORD') }}"
                dbname: "{{ secret('DBT_DBNAME') }}"
                schema: analytics
                threads: 4

triggers:
  - id: daily_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 8 * * *"

errors:
  - id: send_slack_alert
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": "dbt pipeline {{ flow.namespace }}.{{ flow.id }} failed, execution {{ execution.id }}."
      }

A few things are worth noticing in this workflow:

• Declarative & Version-Controlled: The entire pipeline is a single YAML file that can be stored in Git alongside your dbt project, ensuring a single source of truth.
• State-Aware Execution: loadManifest restores the previous run’s manifest.json from the KV Store and storeManifest saves the new one, so dbt build --select state:modified+ with --defer rebuilds only the models that changed, saving time and compute.
• Shared Working Directory: The clone and the dbt build sit inside a WorkingDirectory task, so dbt sees the files the clone just fetched.
• Integrated Error Handling: The errors block runs only when a task fails, and sends the flow and execution identifiers to Slack without extra code.
• Isolated Environment: The containerImage property ensures that dbt runs in a specific, containerized environment, guaranteeing consistency and avoiding dependency conflicts.

Choosing your dbt orchestration strategy

The right tool for dbt orchestration depends on your team’s needs.

• dbt Cloud Scheduler: For teams whose workflows live entirely within dbt and require simple, time-based scheduling, the native dbt Cloud scheduler is a convenient and well-integrated option.
• External Orchestrator (Kestra, Airflow): For teams that need to integrate dbt with the rest of their stack, an external orchestrator is essential. This is the case when you need to trigger dbt runs from upstream tools, manage complex dependencies with non-dbt systems (like Databricks Workflows), or require advanced error handling and event-driven capabilities. Comparing tools like Airflow vs Kestra often comes down to preferring declarative YAML over Python scripting and seeking a more language-agnostic platform.

Where dbt orchestration pays off

Implementing a dedicated orchestration layer for your dbt projects delivers significant returns:

• Enhanced Reliability: With automated retries, dependency checks, and proactive alerting, pipelines become more resilient, reducing manual intervention and firefighting.
• Cost Optimization: State-aware and event-driven runs prevent unnecessary computation in the data warehouse, directly lowering costs.
• Improved Data Quality: Orchestration allows you to embed data quality tests and validation steps directly into your pipelines, catching issues before they impact downstream consumers.
• Faster Development Cycles: By automating the testing and deployment process, teams can ship changes to dbt projects faster and more confidently.
• Unified Observability: A central orchestrator provides a single place for data pipeline monitoring and workflow observability, making it easier to debug issues across the entire data stack.

Related concepts

• What Is dbt in Data Engineering?
• dbt Integrations for Data Orchestration
• dbt on Airflow vs Kestra: Orchestration Comparison
• ETL Workflow Explained
• GitOps for Data and Infrastructure Pipelines
• What Is Data Orchestration?