Publish Pipeline Completion Events to Pub/Sub

Turn a finished pipeline into an event other systems can react to. This blueprint runs a scheduled Google BigQuery aggregation, then publishes a structured completion event (run id, status, row count) to a Google Cloud Pub/Sub topic. Instead of polling for "is the pipeline done yet?", downstream services and other Kestra flows subscribe to the topic and act the moment the event lands. It solves the classic event-driven orchestration gap: a batch job finishes, but nothing downstream knows until someone wires up a fragile cron offset or a manual handoff.

How it works

1. A io.kestra.plugin.core.trigger.Schedule trigger fires on a cron cadence (default 0 7 * * *, shipped disabled so you can review it first).
2. The run_aggregation task (io.kestra.plugin.gcp.bigquery.Query) runs the pipeline's SQL with fetchOne: true and captures a row_count.
3. The publish_event task (io.kestra.plugin.gcp.pubsub.Publish) publishes a message to the pipeline-events topic using serdeType: JSON. The data object carries run_id, status, and row_count, while attributes (source, flow) ride alongside for subscriber-side filtering.
4. The notify task (io.kestra.plugin.email.MailSend) emails the data team to confirm the publish.
5. An errors handler (alert_on_failure) emails the data team if the publish fails.

GCP authentication is centralized through pluginDefaults, injecting projectId and serviceAccount into every io.kestra.plugin.gcp task.

What you get

• A reusable event emitter that fans out a single pipeline completion to many consumers.
• Structured JSON payloads plus message attributes ready for Pub/Sub subscription filters.
• Email confirmation on success and automatic alerting on failure.
• Centralized GCP credentials via pluginDefaults.

Who it's for

• Data engineers decoupling batch pipelines from downstream consumers.
• Platform teams building event-driven architectures on Google Cloud.
• Analytics teams that need other flows or services to trigger off completed jobs.

Why orchestrate this with Kestra

Pub/Sub moves messages but has no scheduler, no aggregation step, and no notion of a pipeline run. Kestra supplies the missing orchestration layer: an event-driven Schedule trigger, automatic retries on transient publish failures, end-to-end execution lineage from query to publish to notification, and a fully declarative YAML definition you can version and review. The completion event becomes a first-class, observable step rather than a side effect buried in application code.

Prerequisites

• A Google Cloud project with a Pub/Sub topic named pipeline-events and a source table analytics.daily_orders.
• A service account with Pub/Sub publish and BigQuery job permissions.
• SMTP credentials for email notifications.

Secrets

• GCP_PROJECT_ID: GCP project hosting the Pub/Sub topic and BigQuery dataset.
• GCP_SERVICE_ACCOUNT: service account key JSON with Pub/Sub publish and BigQuery access.
• SMTP_HOST: SMTP server host.
• SMTP_USERNAME: SMTP account username.
• SMTP_PASSWORD: SMTP account password.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Point the topic and the BigQuery sql at your own resources.
3. Enable the daily Schedule trigger (it ships disabled: true) and adjust the cron.
4. Run the flow once manually to confirm the event publishes and the email arrives.

How to extend

• Reshape the data object and attributes to match what your consumers expect.
• Subscribe other Kestra flows to pipeline-events with the Pub/Sub trigger to chain pipelines.
• Add subscription filters on attributes so each consumer only sees relevant events.
• Swap the BigQuery step for any upstream task and keep the publish pattern unchanged.

Links

• Plugin docs: https://kestra.io/plugins/plugin-gcp
• Orchestrate Pub/Sub with Kestra: https://kestra.io/orchestration/pubsub