ADF Pipeline Explained: Concepts, Capabilities, and Orchestration with Kestra

TL;DR — An Azure Data Factory (ADF) pipeline is a logical grouping of activities that automates data movement and transformation processes within the Azure cloud. It acts as the core component for building ETL/ELT workflows, connecting various data sources to destinations.

Azure Data Factory (ADF) pipelines are the backbone of many cloud-native data strategies, enabling organizations to move and transform data within the Azure ecosystem. While powerful for Azure-centric workloads, managing complex dependencies, diverse technologies, and hybrid cloud scenarios with a single cloud provider’s tool can introduce operational overhead.

This article demystifies ADF pipelines, explaining their core components and how they facilitate ETL. We’ll explore their capabilities and limitations, and then demonstrate how Kestra can provide a unified, declarative orchestration layer to coordinate ADF pipelines alongside your entire data, AI, and infrastructure stack, even across hybrid and multi-cloud environments.

How ADF Pipelines Work: Core Concepts

An ADF pipeline is a workflow composed of one or more activities that perform a specific task. To understand how they function, it’s essential to grasp their core components:

• Pipeline: The top-level resource that logically groups a set of activities. It defines the workflow’s structure and execution flow.
• Activities: These are the individual processing steps within a pipeline. Common activities include Copy Data for moving data between stores, Data Flow for visual data transformation, Stored Procedure for invoking database logic, and Web Activity for calling REST APIs.
• Datasets: A named view of data that points to or references the data you want to use in your activities as inputs or outputs. Datasets define the structure of the data within the connected data stores.
• Linked Services: Similar to connection strings, linked services define the connection information needed for Data Factory to connect to external resources. This could be an Azure Blob Storage account, an on-premises SQL Server, or a SaaS application.
• Triggers: These initiate the execution of a pipeline. ADF supports several trigger types, including Schedule for time-based execution, Tumbling Window for processing time-sliced data, and Event-based triggers that respond to events like file arrival in Azure Blob Storage.

These components work together to create a cohesive data pipeline. A trigger fires, executing a pipeline. The pipeline runs its activities in a defined sequence, using linked services to connect to datasets that represent the source and destination of the data.

Why ADF Pipelines Need Robust Orchestration

While ADF is a capable tool for Azure-native data integration, production environments often demand orchestration capabilities that span beyond a single cloud service. Relying solely on ADF’s internal scheduler can create challenges:

• Complex Dependencies: Real-world workflows often involve dependencies on services outside of Azure, such as on-premises databases, other cloud providers, or third-party SaaS applications. Managing these cross-platform dependencies within ADF can be complex.
• Error Recovery and Alerting: While ADF has retry mechanisms, implementing sophisticated, multi-step error recovery logic that involves external systems requires a more powerful orchestration layer.
• End-to-End Visibility: Gaining a unified view of a workflow that includes an ADF pipeline, a dbt transformation, and a notification to a Slack channel is difficult without a central orchestrator.
• Hybrid and Multi-Cloud Scenarios: Organizations often operate in hybrid or multi-cloud environments. An orchestration tool locked into a single cloud ecosystem limits flexibility and increases operational complexity.
• CI/CD and GitOps: Implementing robust CI/CD practices for data pipelines requires a declarative, version-controlled approach to workflow definitions, which is where a platform like Kestra excels. You can manage your data pipelines as code and integrate them into your existing DevOps toolchain.

Orchestrate ADF Pipelines with Kestra: An End-to-End Example

Kestra acts as a universal control plane that can trigger, monitor, and manage ADF pipelines as part of a larger, technology-agnostic workflow. Instead of being limited to Azure’s ecosystem, you can coordinate ADF with any tool or service that has an API or CLI.

The following Kestra flow demonstrates how to trigger an ADF pipeline, poll for its completion status, and send a Slack notification based on the outcome. This entire process is defined in a single, declarative YAML file.

Orchestrating ADF with the Azure CLI in Kestra

One of the most flexible ways to interact with Azure services is through the Azure CLI. Kestra’s Commands task can execute any shell command within a containerized environment, providing a universal interface to manage services like ADF. This approach allows you to leverage your existing knowledge of the Azure CLI and integrate it seamlessly into a governed, observable workflow.

id: trigger-and-monitor-adf-pipeline
namespace: prod.azure
description: Triggers an Azure Data Factory pipeline, polls for its status, and sends notifications.

tasks:
  - id: install-azure-cli
    type: io.kestra.plugin.scripts.shell.Commands
    description: "Ensure Azure CLI is available in the runner environment"
    runner: DOCKER
    docker:
      image: mcr.microsoft.com/azure-cli:latest
    commands:
      - az --version

  - id: trigger-adf-pipeline
    type: io.kestra.plugin.scripts.shell.Commands
    description: "Start the ADF pipeline run and capture the run ID"
    runner: DOCKER
    docker:
      image: mcr.microsoft.com/azure-cli:latest
    env:
      AZURE_CLIENT_ID: "{{ secret('AZURE_CLIENT_ID') }}"
      AZURE_CLIENT_SECRET: "{{ secret('AZURE_CLIENT_SECRET') }}"
      AZURE_TENANT_ID: "{{ secret('AZURE_TENANT_ID') }}"
    commands:
      - az login --service-principal -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET --tenant $AZURE_TENANT_ID
      - |
        run_id=$(az datafactory pipeline create-run \
          --factory-name "YourADFName" \
          --name "YourPipelineName" \
          --resource-group "YourResourceGroup" \
          --query "runId" -o tsv)
        echo "::> runId: $run_id"

  - id: poll-pipeline-status
    type: io.kestra.plugin.scripts.shell.Commands
    description: "Poll the ADF pipeline run status until it completes"
    runner: DOCKER
    docker:
      image: mcr.microsoft.com/azure-cli:latest
    env:
      AZURE_CLIENT_ID: "{{ secret('AZURE_CLIENT_ID') }}"
      AZURE_CLIENT_SECRET: "{{ secret('AZURE_CLIENT_SECRET') }}"
      AZURE_TENANT_ID: "{{ secret('AZURE_TENANT_ID') }}"
    retry:
      type: constant
      maxAttempts: 20
      interval: PT1M
    commands:
      - az login --service-principal -u $AZURE_CLIENT_ID -p $AZURE_CLIENT_SECRET --tenant $AZURE_TENANT_ID
      - |
        status=$(az datafactory pipeline-run show \
          --factory-name "YourADFName" \
          --resource-group "YourResourceGroup" \
          --run-id "{{ outputs['trigger-adf-pipeline'].vars.runId }}" \
          --query "status" -o tsv)

        if [ "$status" == "Succeeded" ]; then
          echo "Pipeline Succeeded"
          exit 0
        elif [ "$status" == "InProgress" ] || [ "$status" == "Queued" ]; then
          echo "Pipeline still running with status: $status"
          exit 1
        else
          echo "Pipeline failed with status: $status"
          exit 2
        fi

  - id: success-notification
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": "ADF Pipeline `{{ flow.namespace }}.{{ flow.id }}` completed successfully. Execution ID: `{{ execution.id }}`"
      }

errors:
  - id: failure-notification
    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": "ADF Pipeline `{{ flow.namespace }}.{{ flow.id }}` failed on task `{{ taskrun.taskId }}`. Execution ID: `{{ execution.id }}`"
      }

What makes this Kestra workflow powerful:

• Declarative & Version-Controlled: The entire end-to-end process is defined in a single YAML file, which can be stored in Git, reviewed, and versioned like any other piece of code.
• Stateful Retries: The retry block on the polling task automatically handles the waiting logic, making the workflow resilient to long-running ADF jobs without complex scripting.
• Unified Notifications: Success and failure notifications are built into the workflow, providing immediate feedback without requiring separate monitoring configurations in Azure.
• Cross-System Coordination: This flow could easily be extended to include tasks that run before or after the ADF pipeline, such as triggering a dbt Cloud job, loading data from an S3 bucket, or updating a Salesforce record.

ADF vs. Microsoft Fabric Data Pipelines: Key Differences

With the launch of Microsoft Fabric, a common question is how ADF pipelines differ from the new Fabric Data Pipelines. While they share a similar purpose, their context and integration are distinct.

ADF is a standalone, mature Azure service focused purely on data integration and ETL. It’s a versatile tool designed to connect a wide array of data sources and destinations.

In contrast, Fabric Data Pipelines are an integrated component of the broader Microsoft Fabric platform. Fabric aims to be an all-in-one analytics solution, unifying data engineering, data science, and business intelligence. Fabric pipelines are designed to work seamlessly with other Fabric components like OneLake, Notebooks, and Power BI. For organizations fully committed to the Microsoft Fabric ecosystem, its pipelines offer a more cohesive user experience. For those needing a more flexible, standalone data integration service that connects to a heterogeneous stack, ADF remains a strong choice. You can find more on Microsoft Fabric alternatives here.

When Kestra Extends ADF for Hybrid and Multi-Cloud Workflows

The true power of a universal orchestrator like Kestra becomes apparent in complex enterprise environments. While ADF excels within its Azure boundaries, Kestra provides the control plane to manage workflows that cross those boundaries.

Consider these scenarios where Kestra extends ADF’s capabilities:

• Hybrid Cloud: An on-premises SQL Server database needs to be ingested into Azure Synapse via an ADF pipeline. Kestra can orchestrate the entire flow, first running a script on the on-premises server to prepare the data, then triggering the ADF pipeline, and finally launching a validation check.
• Multi-Cloud: A workflow might start by pulling data from Google Cloud Storage, processing it with an ADF pipeline, and then loading the results into a Snowflake warehouse running on AWS. Kestra can manage this entire multi-cloud process from a single point of control.
• Beyond ETL: Data pipelines are often part of a larger business process. Kestra can orchestrate ADF as part of an infrastructure automation workflow that also involves provisioning resources with Terraform or running Ansible playbooks.

By using Kestra, you are not replacing ADF; you are elevating it. ADF handles the Azure-native data movement, while Kestra provides the overarching orchestration, observability, and governance across your entire technology stack.

Related Concepts

• Best ETL Pipeline Tools for Data Engineering
• SSIS Replacements: Modern ETL & Orchestration
• Top Free ETL Tools 2026: Open Source & Kestra Alternatives
• DataStage Migration: Strategies & Modern Cloud ETL
• Top Matillion Alternatives for Data Integration
• Talend Alternatives: Top Data Integration Tools (2026)

Explore Kestra’s capabilities for unified data orchestration.