Cloud Composer gives GCP-native teams managed Apache Airflow without running the infrastructure themselves. Kestra takes a different approach: declarative YAML orchestration that runs on any cloud, in any language, covering data pipelines, infrastructure automation, and business processes, without a 25-minute environment provisioning step before your first workflow runs.
Declarative YAML orchestration with no cloud dependency baked into the architecture. Workflows describe what should run, not which managed services to wire together first. Python, SQL, Bash, R, and Go tasks run in isolated containers alongside GCP-native tasks, on any cloud or on-prem.
Apache Airflow, fully managed on Google Kubernetes Engine. Cloud Composer handles Airflow upgrades, GKE cluster management, and Cloud SQL provisioning. DAGs are stored in Cloud Storage, and GCP IAM controls access. The GCP integration runs deep: BigQuery, Dataflow, Cloud Run, and Pub/Sub are first-class targets, and Python is the only way to author them.
Cloud Composer is Google's fully managed Airflow service—there is no local or self-hosted install option. This comparison reflects what's required to provision a Composer environment: enabling the API, creating a service account with Composer Worker permissions, configuring IAM roles across five permission levels, and waiting roughly 25 minutes for the GKE-backed environment to provision.
curl -o docker-compose.yml \https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.ymldocker compose up
# Open localhost:8080# Pick a Blueprint, run it. Done.Download the Docker Compose file, spin it up, and you're ready (database and config included). Open the UI, pick a Blueprint, run it. No GCP project, no service account, no IAM roles, no GCS bucket.
Enable the Cloud Composer API, create a service account with Composer Worker role, assign five IAM permission levels, create the Composer environment (~25 min to provision), then upload DAG files to the GCS bucket and wait for the Airflow scheduler to detect them.
YAML is readable on day 1. Our docs are embedded in the UI for easy reference, the AI Copilot writes workflows for you, or start with our library of Blueprints. No Python knowledge required to understand or modify a workflow.
DAG files are Python scripts stored in a GCS bucket. Updating a workflow means pushing a new file to Cloud Storage and waiting for the Airflow scheduler to detect the change. Operators, dependencies, and scheduling logic are all Python. Non-Python contributors can view runs in the Airflow UI but cannot author or edit workflows.
Orchestrate across data pipelines, infrastructure operations, business processes, and AI workflows in one unified platform. Event-driven at its core, with native triggers for Pub/Sub, GCS file arrivals, webhooks, Kafka, and database changes. Runs on any cloud or on-prem.
Managed Airflow on GKE with first-class GCP integration. Excellent for Python-based data pipelines that target BigQuery, Dataflow, and Cloud Run. Infrastructure automation and business process workflows are possible via Python operators but are not first-class use cases. Runs exclusively on GCP.
| | | |
|---|---|---|
| Workflow definition | Declarative YAML | Python DAGs stored in Cloud Storage |
| Languages supported | Any (Python, SQL, R, Bash, Go, Node.js) | Python-first (Bash/SQL via operators) |
| Cloud dependency | Runs anywhere (GCP, AWS, Azure, on-prem) | GCP only |
| Setup requirements | Docker Compose (two commands) | API enable, IAM roles, service account, GKE environment (~45 min) |
| Architecture | Event-driven at core | Schedule-first (Airflow under the hood) |
| Airflow version control | Your upgrade timeline | Google controls Airflow version availability and cadence |
| Multi-tenancy | Namespace isolation + RBAC out-of-box | IAM-based access; separate Composer environments for strong isolation |
| Self-service for non-engineers | Kestra Apps | Airflow UI is observability-focused, no self-service authoring |
| Infrastructure automation | Native support | Possible via Python operators and GCP SDK, not a primary use case |
We switched from Airflow because we want engineers solving problems, not coding orchestration. Kestra delivers end-to-end automation with the robustness we need at our scale. Few companies operate at this level, especially in AI/ML.
Kestra runs in two Docker Compose commands. No GCP project to configure, no service account to provision, no IAM roles to assign across five permission levels, no 25-minute environment creation window. Teams get a production-shaped Kestra instance locally in under 5 minutes, using the same Docker-based deployment that scales to Kubernetes in production.
When your pipelines pull from BigQuery but push to Snowflake or trigger infrastructure jobs on AWS, routing everything through a GCP-only control plane adds friction. Kestra runs on GCP, AWS, Azure, or your own data center, with native plugins for GCS, BigQuery, Pub/Sub, and every major GCP service alongside AWS and Azure targets.
Kestra upgrades run on your schedule: pull a new image, test in staging, and ship when ready. No waiting for a managed service to certify a release or schedule a maintenance window. Roll back instantly if something unexpected surfaces after an upgrade.
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
See how Kestra can simplify your data pipelines—and run them on any cloud, not just GCP.