Query icon
If icon
PySparkSubmit icon
DiscordIncomingWebhook icon
gcp icon
Schedule icon

Gate a Dataproc PySpark Batch on Data Readiness

Orchestrate Dataproc Serverless with Kestra. Check BigQuery for fresh source rows, then submit a PySpark batch only when the data has landed.

Categories
CloudData

More Cloud Blueprints

High-Performance Computing, Run Parallel Python Workloads on AWS Batch with ECS Fargate

Provision AWS Batch on ECS Fargate with Terraform, then run parallel Python workloads on it from one Kestra flow.

WorkingDirectory icon
Clone icon
TerraformCLI icon
Script icon
Docker icon
Parallel icon
+2
Cloud Core Infrastructure
Enterprise Edition
Autoscale EMR Serverless Spark on a CloudWatch Backlog

Watch a CloudWatch backlog metric and fire an EMR Serverless Spark job only when work is pending.

Query icon
If icon
StartServerlessJobRun icon
SlackIncomingWebhook icon
Schedule icon
Cloud Data
Publish Application and System Events to AWS EventBridge for Event-Driven Architectures

Publish custom application and system events to AWS EventBridge using inline definitions or bulk JSON payloads.

PutEvents icon
Download icon
JsonToIon icon
Cloud

More Data Blueprints

Run a Scheduled ETL Sync with Airbyte Cloud

Trigger an Airbyte Cloud connection sync on a recurring schedule so your data lands fresh without manual runs.

Sync icon
Schedule icon
Data
Sync Postgres records to Algolia index

Incrementally push freshly updated Postgres rows into an Algolia search index every five minutes.

Trigger icon
ForEach icon
Index icon
Query icon
postgresql icon
Business Data
Run Parallel ETL Ingestion with Airbyte to Sync Multiple Data Sources

Trigger multiple Airbyte connection syncs in parallel from a single Kestra flow to cut total ingestion time.

Parallel icon
Sync icon
Data
See How

New to Kestra?

Use blueprints to kickstart your first workflows.

Get Started