Postgres stores and queries the data. Kestra decides when each job runs, what feeds it, and what runs next: bulk-load files the moment they land, run transactional multi-statement loads, react to row changes with CDC, and trace every statement in one execution history.
pg_cron schedules SQL inside one database on a clock. Kestra runs everything around Postgres: it downloads a file and bulk-loads it with COPY the moment it arrives, chains Postgres with dbt transforms and S3 in one flow, runs multi-statement loads as a single transaction so a partial batch never lands, and reacts to row changes with Debezium CDC. Every run gets retries, backfills, and a logged history that pg_cron cannot give you.
Postgres runs the storage, the queries, and the COPY. Kestra runs the steps around the query: what triggers it, what feeds it, what runs next, and where the cross-tool audit trail lives.
pg_cron fires SQL inside one database on a fixed clock. If the upstream extract slips, it runs on stale data. Kestra triggers io.kestra.plugin.jdbc.postgresql.Query the moment the upstream step confirms, whether that is a file landing in S3, a webhook, or an Airbyte sync. The Trigger task also polls Postgres and starts a flow when a query returns rows.
Loading a file means download it, create the table, COPY it in, then validate the count. Kestra runs core.http.Download, then io.kestra.plugin.jdbc.postgresql.CopyIn over the COPY protocol (the fastest path in Postgres), then a row check as ordered steps with per-step retries. A failed COPY retries without re-downloading the file.
An incremental load is rarely one statement: stage the rows, MERGE into the target, dedup, update a watermark. io.kestra.plugin.jdbc.postgresql.Queries runs them as one transaction, so a failure rolls back and a half-applied batch never lands. The next run resumes cleanly from the last committed watermark.
Postgres has no view of the connector that fed it or the dbt models that read it. Kestra runs ingestion, the Postgres load, the transform, and the reverse-ETL push as steps in one flow. Run IDs flow forward and the whole chain shares one execution ID, so you debug load, transform, and sync from one screen, not three.
Time-based scheduling cannot react to a single changed row. io.kestra.plugin.debezium.postgres.RealtimeTrigger reads the Postgres write-ahead log and starts one execution per change, so inserts and updates fan out to downstream systems like Kafka the instant they commit, with no polling lag and no missed rows.
Analysts should not need database credentials to run a parameterized export. Kestra Apps wraps a Query flow in a typed form: pick the date range, run the query, get the result file. Every run lands in execution history with the requesting user, so self-service does not mean unaudited.
Patterns data engineering teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
A CSV lands or an API exposes a dataset. Kestra downloads it, runs a Query to create the table if needed, then bulk-loads it with CopyIn over the COPY protocol. A row-count check gates anything downstream, and a failed load retries without re-downloading the file.
CopyIn streams the file in one operation, the fastest load path in Postgres.
An If branches on the loaded row count before anything reads the table.
A COPY failure retries on its own without re-running the download.
One source, two sinks. Kestra downloads the data, transforms it in Python, then writes it in parallel to Postgres with CopyIn and to S3 as a backup. Each write is isolated, so a slow S3 upload never blocks the database load, and an email confirms with the row count.
A single Python step shapes the rows for both Postgres and the S3 archive.
Each sink writes independently, so one slow target never blocks the other.
Extract, transform, and a dual load finish in one short execution.
Models live in version control, not in the database. Kestra clones the dbt project, runs dbt build against Postgres in a container, then posts to Microsoft Teams. A failing test stops the chain before downstream consumers read half-built tables.
git.Clone pulls the exact committed dbt project before every build.
A failing dbt test halts the flow before consumers read the tables.
Point the same flow at Snowflake or Redshift by changing the dbt profile.
Some pipelines react to data, not clocks. The Debezium Trigger reads the Postgres write-ahead log and starts a flow the moment rows change, posts the change count to Slack, then processes the captured payload in Python. No polling lag, no missed commits.
Debezium streams inserts, updates, and deletes straight from the write-ahead log.
Each committed change set starts its own run with the payload attached.
The captured JSON flows into a Python, shell, or SQL step for downstream work.
Operational data lives in Postgres, but the sales team works in the CRM. The Postgres Trigger polls for changed rows, fans them out with ForEach, and creates or updates each record in HubSpot, so the CRM stays current without a nightly batch and a Discord ping confirms the run.
The Trigger query returns new or updated records, not the whole table.
ForEach creates or updates each row in the destination independently.
Swap the HubSpot task for Salesforce, Slack, or any API target.
Our goal is for Kestra to become the go-to solution across the organization whenever data wrangling and complex pipeline orchestration are needed.
| Capability | pg_cron | |||
|---|---|---|---|---|
| Trigger jobs on upstream completion or events | Native event + flow triggers | Cron inside the database only | Sensor-based polling | Sensors, Python required |
| Move data across systems, not just run SQL | HTTP, S3, dbt, APIs as tasks | SQL in one database only | Operators, Python required | Ops and assets, Python required |
| Bulk load files with COPY | CopyIn and CopyOut tasks | Hand-written SQL only | PythonOperator glue | Custom op |
| Transactional multi-statement load | Queries in one transaction | Hand-written PL/pgSQL | Operator chain | Custom op |
| React to row changes with CDC | Debezium trigger, one run per change | No, time-based only | Custom sensor | Custom sensor |
| Retries, backfills, replay from failure | Per-task, built in | None | Yes, Python config | Yes, Python config |
| Declarative YAML, IaC-friendly | YAML + Terraform provider | SQL DDL | Python DAGs | Python assets |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Bulk-load files with COPY, run transactional loads, react to row changes with CDC, and chain Postgres with dbt and object storage in one flow. Open source, self-hosted, event-driven.
