YAML vs Python Workflows: Which Is Better for Orchestration?

The YAML vs Python debate in workflow orchestration is mostly a layer problem. Python is a general-purpose programming language built for execution logic. YAML is a data serialization format built for expressing structure. Teams that use both well keep them in separate layers: YAML defines what runs and when, Python does the work inside each task.

This is a practical breakdown of where each tool genuinely excels, when to reach for one over the other, and how to use both together. If you want the architectural case for why these layers should be separate in the first place, I covered that in depth in YAML-First Orchestration: Why Workflow Definitions Belong in Config, Not Code.

What each tool actually is

YAML (YAML Ain’t Markup Language) is a data serialization format. It expresses structure: key-value pairs, lists, nested objects. It has no execution model, no control flow, no functions. A YAML file is data that something else reads and acts on. Its value is in being legible to humans and parseable by machines without running anything.

Python is a general-purpose programming language with execution semantics, a runtime, an import system, and a full standard library. It can generate YAML, parse it, and produce dynamic structures programmatically. It can also define orchestration logic, though as I’ll show, that creates coupling problems that YAML avoids by design.

Their apparent similarity (both use indentation) is a surface coincidence that misleads more than it clarifies.

Where Python excels

Python’s strengths in a workflow context cluster around the execution layer: the work that tasks actually do.

Data transformation and ML pipelines

Python’s data ecosystem is unmatched. Pandas, Polars, NumPy, scikit-learn, PyTorch: these libraries exist because Python became the default language for data work, and they’re mature enough that rewriting equivalent logic in any other language would be counterproductive. When a workflow task needs to reshape a DataFrame, train a model, or run a statistical test, Python is the right tool.

The key is that this Python belongs inside a task, not wrapped around the workflow definition. The orchestrator should coordinate when the transformation runs and what happens if it fails; the Python script should do the transformation. For a closer look at where Python fits relative to orchestration logic, Benoit’s post on Python for business logic is worth reading alongside this one.

Dynamic configuration generation

YAML is static by nature. When the configuration itself needs to be computed — generating dozens of workflow variants for different regions, customers, or tables; templating YAML with Jinja2; building configuration from database queries at runtime — Python does work that YAML can’t.

This pattern is common in data platform engineering: a Python script generates a family of YAML files, each representing a workflow variant. The generation logic stays in Python; the resulting workflows are expressed in YAML and run independently. Python produces the config; YAML is the config.

Custom integrations and complex business logic

When a workflow task needs to handle a third-party API with non-standard authentication, implement custom retry logic beyond what an orchestrator provides natively, or process data through a pipeline where branching depends on runtime state, Python is the right choice. The control flow expressiveness that makes Python verbose for simple orchestration makes it powerful for intricate execution logic.

The pattern that works: define the workflow in YAML, implement the complex logic in a Python script, and have the YAML workflow call the script as an external process. The orchestrator coordinates; Python executes.

Where YAML excels

YAML’s strengths cluster around the coordination layer: defining what should run, when, in what order, and under what conditions.

Readability without a runtime

A YAML workflow is legible to anyone who can read text. There’s no decorator pattern to unwrap, no implicit registration to trace, no mental model of how the framework interprets module-level function calls. The analytics engineer who writes SQL and the platform engineer who manages infrastructure can both read a YAML workflow without a Python environment or framework knowledge.

Workflows outlive the people who wrote them. A format that’s readable without setup survives oncall rotations, team changes, and cross-functional ownership in a way that framework-specific Python code doesn’t.

GitOps and auditability

When a task is added, a dependency changes, or a schedule shifts, a YAML diff shows exactly what moved. Code review is legible to anyone on the team, not just whoever wrote the pipeline. That makes YAML the natural fit for GitOps practices: changes go through pull requests, history is auditable, rollback is a revert.

Python DAGs are harder to diff meaningfully because orchestration intent is mixed with execution code. Adding a task might touch imports, decorator parameters, and operator wiring in ways that obscure what actually changed about the workflow.

Accessibility across roles

Python-based orchestration restricts workflow ownership to people who know both Python and the framework’s specific abstractions. YAML sidesteps both requirements. As orchestration scope expands beyond data pipelines to infrastructure automation, API coordination, and business processes, the relevant audience is no longer a single-language community.

For the full architectural argument on why this matters, see YAML-First Orchestration.

Portability and environment management

A YAML workflow definition is portable by nature. The same file runs across dev, staging, and production without modification. It has no dependency on a Python version, a virtual environment, or a specific framework install. Anyone with access to the orchestrator can run it.

Python-based workflow definitions carry environment state. The framework version matters, the import structure matters, and dependency conflicts between orchestration packages and task packages are a real operational burden. Teams working across environments often end up managing separate requirements files for orchestration code and execution code — a symptom of the coupling problem.

Kestra sidesteps this at both layers: the YAML definition has no runtime dependencies, and each task declares its own container image. A task that needs Polars 0.19 and a task that needs NumPy 1.26 run in separate containers with no conflict. The orchestration layer stays clean; dependency management moves to where it belongs.

AI generation

YAML is also far easier for AI to generate correctly. Every task’s type, properties, and relationships are visible in the structure itself. An LLM can validate output against a schema without executing anything. Python DAGs, with their decorators and implicit registration conventions, are harder for AI to generate correctly because the semantics aren’t visible in the syntax.

YAML vs JSON: different roles, not competitors

A common question alongside the YAML vs Python comparison is whether YAML is competing with JSON. It isn’t.

JSON is the right format for data interchange: APIs return JSON, SDKs serialize to JSON, databases store JSON. Its strict syntax (no comments, no trailing commas, no multi-line strings) makes it unambiguous to parse and reliable for machine-to-machine communication.

YAML is the right format for human-readable configuration. Comments, multi-line strings, and a less rigid syntax make it easier to write and review. Those same features make it less suitable for data interchange, where strictness matters more than readability.

In practice: your workflow definitions live in YAML; the data flowing between your tasks might be JSON. They’re complementary at different points in the stack.

The learning curve

YAML’s cognitive overhead for workflow definition is low. Five concepts cover nearly everything: key-value pairs, nesting via indentation, lists, multi-line strings, and comments. Someone who has never written YAML before can typically read a workflow definition and understand what it does without documentation.

The most common parse errors are tabs instead of spaces, a missing space after a colon, and unquoted special characters. YAML 1.1 also parses NO, yes, on, and off as booleans, so country codes and feature flags need quotes. For a full reference, the YAML basics guide covers the common pitfalls.

Python’s learning curve for workflow definition is steeper, because you’re learning Python the language plus the orchestrator’s abstractions (DAG classes, decorators, operator patterns) plus the implicit conventions that make those abstractions work together. For someone coming from SQL or a scripting background, that’s a real barrier.

For beginners, YAML’s constraint is actually an advantage: you can’t accidentally put business logic in your workflow definition because YAML has no execution semantics. The logic lives in scripts and tasks, where it belongs.

Where YAML hits its limits

YAML’s simplicity becomes a constraint when workflow definitions grow very large. A single file defining dozens of tasks with deeply nested conditional logic is harder to maintain than equivalent modular Python. YAML has no functions, imports, or abstractions for reuse.

The practical response: keep individual workflow definitions focused. When you find yourself duplicating large blocks of YAML across workflows, that’s a signal to extract shared logic into a reusable subflow, not to switch from YAML to Python for workflow definition. Kestra’s subflows handle most reuse cases without giving up declarative structure. The YAML pitfalls post covers the common parse errors and edge cases worth knowing before you hit them.

Practical guide: when to reach for each

Reach for YAML when:

• Defining what runs, when, in what order, and under what conditions
• Your team includes roles beyond Python developers (analysts, platform engineers, data scientists)
• Workflow changes need to be reviewable in pull requests without framework knowledge
• You want AI to generate or modify workflows reliably

Reach for Python when:

• A task involves data transformation, ML, or statistical computation
• You need to generate workflow definitions programmatically (Jinja2 templating, dynamic variants)
• Business logic requires complex runtime branching or custom integration behavior
• You’re building execution scripts that the orchestrator will call as external processes

The clearest signal that you’ve mixed these layers incorrectly: your orchestration code and your business logic are in the same files, and changing a schedule requires touching the same code as changing a transformation.

Using both: the Kestra approach

Kestra’s model makes the separation concrete. The workflow below fetches from a REST API, transforms with Python and Polars, then runs SQL analytics with DuckDB — three tools, each in isolation, coordinated by YAML.

Blueprint: REST API to DuckDB with Python and Polars →

id: api-python-sql
namespace: company.team

tasks:
  - id: api
    type: io.kestra.plugin.core.http.Request
    uri: https://dummyjson.com/products

  - id: python
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    containerImage: python:slim
    dependencies:
      - polars
    outputFiles:
      - products.csv
    script: |
      import polars as pl
      data = {{ outputs.api.body | jq('.products') | first }}
      df = pl.from_dicts(data)
      df.glimpse()
      df.select(["brand", "price"]).write_csv("products.csv")

  - id: sql_query
    type: io.kestra.plugin.jdbc.duckdb.Query
    inputFiles:
      in.csv: "{{ outputs.python.outputFiles['products.csv'] }}"
    sql: |
      SELECT brand, round(avg(price), 2) as avg_price
      FROM read_csv_auto('{{ workingDir }}/in.csv', header=True)
      GROUP BY brand
      ORDER BY avg_price DESC;
    fetchType: STORE

The Python script runs in its own Docker container with Polars installed, exactly as it would standalone. The YAML handles task ordering and passes outputs between steps. The SQL query doesn’t know Python ran before it — it just reads the file the orchestrator passed in.

Kestra’s AI Copilot extends this further: describe the workflow in plain language and it generates schema-validated YAML. The AI generates the coordination layer; your team writes the execution scripts. Teams migrating from Python-first orchestrators can compare the approach directly: Kestra vs Airflow and Kestra vs Prefect both cover what changes when you move the workflow definition out of Python.

Where to go from here

• Read the architectural case. YAML-First Orchestration makes the argument for why separating the coordination layer from the execution layer is the right call, with examples from infrastructure and analytics tooling that already went declarative.
• Browse more workflows. Kestra’s 260+ Blueprints are production-ready YAML workflow templates covering data pipelines, infrastructure automation, and business processes.
• Try Python orchestration. The Python orchestration docs show how Python scripts run as Kestra tasks, with full access to libraries and dependencies.
• Try it live. Install Kestra (one Docker command), open the editor, and paste any example from this post. The visual editor shows the DAG topology updating as you type.
• Why Kestra. For a broader look at the design decisions behind YAML-first orchestration, Why Kestra covers the architectural reasoning.