Home/Blog

Orchestrate Your Airflow Jobs with Kestra: One Workflow at a Time

October 22 2024Solutions

Martin-Pierre Roset

Table of Contents

OVERVIEW

Plugins have a Min. Compatible Kestra Version i.e. a Kestra version from where the corresponding plugin version is compatible. Learn more

Created by

Kestra Core Team

Managed by

Kestra Core Team

Table of Contents

Contribute

Migrating from one orchestration tool to another can seem like an intimidating task—especially if you have critical workflows running in production. When you rely on Airflow for essential data processing, the idea of moving everything to a new platform at once might feel risky. That’s why with Kestra, you don’t have to jump into a big-bang migration. Instead, you can transition one workflow at a time and gradually adopt Kestra’s advanced orchestration capabilities while keeping what works in Airflow.

By allowing you to integrate and manage your existing Airflow DAGs alongside Kestra’s workflows, Kestra provides a unified platform to run, monitor, and orchestrate both old and new systems.

Let’s dive into how you can orchestrate Airflow jobs using Kestra and how this approach helps developers avoid the headaches that come with full-scale migrations.

The Strangler Fig Pattern for Orchestration

This gradual migration is part of a well-known strategy called the Strangler Fig Pattern, where the new system (Kestra) slowly replaces the old one (Airflow) by taking over its workflows, piece by piece. Over time, more and more workflows run in Kestra, while Airflow’s role diminishes—until, eventually, Kestra handles everything.

This approach avoids the risks and complexity of doing a full migration in one go. Instead of uprooting everything at once, you can orchestrate Airflow DAGs within Kestra’s control plane and centralized UI, gaining better visibility and scalability, while continuing to leverage what’s already working in Airflow.

Airflow Plugin: Migrate Without Disruption

In response to many requests from users seeking support for easier migrations from Airflow, we’ve developed a plugin that lets you trigger and orchestrate Airflow DAGs directly from within Kestra. This makes it possible to run Airflow jobs as part of your Kestra workflows, giving you the flexibility to incorporate your existing DAGs into Kestra’s broader orchestration capabilities.

Here’s an example of how you can use Kestra to trigger an Airflow DAG:

id: airflow
namespace: company.team

tasks:
  - id: run_dag
    type: io.kestra.plugin.airflow.dags.TriggerDagRun
    baseUrl: http://host.docker.internal:8080
    dagId: hello_world_dag
    wait: true
    pollFrequency: PT1S
    options:
      basicAuthUser: "{{ secret('AIRFLOW_USERNAME') }}"
      basicAuthPassword: "{{ secret('AIRFLOW_PASSWORD') }}"
    body:
      conf:
        source: kestra
        namespace: "{{ flow.namespace }}"
        flow: "{{ flow.id }}"
        task: "{{ task.id }}"
        execution: "{{ execution.id }}"

In this setup:

• Trigger Airflow DAGs through Kestra’s Airflow plugin using the Airflow REST API.
• Monitor and poll the status of your Airflow tasks directly within Kestra, allowing for real-time visibility.
• Pass execution metadata (like task and flow IDs) to maintain context and track workflow performance across both platforms.

As we can see, Kestra catch all the dag run information.

On the other side, the Airflow DAG is triggered successfully.

Kestra: A Central tool for All Your Workflows

Once integrated, Kestra becomes the central control plane for orchestrating workflows across your stack. Whether it’s managing complex real-time data pipelines or orchestrating legacy Airflow jobs, you can monitor all executions through Kestra’s dashboard, which offers deeper insights and enhanced monitoring compared to Airflow’s built-in tools. With centralized logging, real-time outputs, and intuitive error tracking, Kestra simplifies your workflow management.

Kestra’s declarative approach makes it easier to build and manage workflows. Say goodbye to the complexity of Python-based DAGs. Instead of managing dependencies, glue code, and intricate DAG structures, Kestra lets you define workflows in a simple, readable format and manage them directly through the UI.

Simplifying Complex Workflows: Get Rid of Glue Code

Airflow is known for its complexity in constructing DAGs—especially when basic workflows end up requiring complicated Python scripts. With Kestra, you can streamline your workflows with a declarative syntax, eliminating the need for glue code and additional scripts.

Here’s how Kestra helps you:

• No need for Python glue code: Kestra’s pre-built tasks handle common operations like HTTP requests, file transfers, and API calls without extra scripts.
• Unified orchestration: Use Kestra to orchestrate tasks across diverse platforms—cloud services, data processing, APIs—within the same workflow.
• UI-based or as code management: Build, trigger, and monitor workflows directly from Kestra’s UI or build everything as code.

What’s Next on the Roadmap?

Currently, you can orchestrate Airflow DAGs using Kestra, but we’re working on expanding this integration. Soon, we’ll provide more detailed documentation and tools to help users migrate Airflow workflows directly into Kestra. The goal is to make it as seamless as possible to shift your orchestration to Kestra at your own pace.

How Can We Help?

We want to hear from you! If there are specific features or tools you’d like to see to support your migration from Airflow, let us know. We’re constantly working on ways to make this transition easier for you.

Conclusion: Migrate Without the Big Bang

Migrating to a new orchestration platform doesn’t have to mean ripping out everything at once. With Kestra, you can adopt a gradual migration strategy, integrating your existing Airflow workflows while gaining access to the advanced orchestration features that Kestra offers. Whether you need a unified UI, better monitoring, or scalable workflows, Kestra simplifies orchestration without the need for complex migrations.

So why not give it a try? Use Kestra to orchestrate Airflow alongside your other workflows and scale at your own pace.

Need to talk about migration just reach out to us we would be happy to discuss this with you!