480+ Blueprints, ready to build.

The first step is always the hardest. Search or browse to kick-start your next flow.

Spark & Big Data

8 Blueprints

Sort by:

Submit a Python Apache Beam pipeline to a distributed Flink portable runner via a JobServer, triggered after upstream ingestion succeeds.

Use the Beam YAML framework with the Python SDK to read, filter, and rewrite a daily orders CSV, backfill-safe and with a non-injectable threshold.

Run the canonical Apache Beam word-count pipeline locally with the DIRECT runner over a real ingested text file.

Watch a CloudWatch backlog metric and fire an EMR Serverless Spark job only when work is pending.

Clone a Git repository of PySpark code and run a Spark job with spark-submit, fully orchestrated by Kestra.

Check BigQuery for fresh source rows and submit a Dataproc Serverless PySpark batch only when the data is ready.

Orchestrate the GCS to Dataproc to BigQuery lakehouse chain in one flow, with a serverless PySpark transform and a gold table load.

Submit a Spark job to Dataproc Serverless on a schedule, with no cluster to manage and cost that tracks the work.