Pulsar Topic Activity Sentinel
Non-destructively sample a Pulsar topic on a schedule with a non-durable reader and page Slack when it goes silent, floods, or the broker is unreachable.
Refresh only changed BigQuery partitions by loading each day's GCS files into its matching partition with WRITE_TRUNCATE, so cost tracks the delta.
Reprocessing an entire BigQuery table every run is slow and expensive when only the last few days of data actually change. This blueprint runs a partition-aware incremental refresh: it walks a small rolling window of days and, for each day, loads just that day's Google Cloud Storage files into the matching date partition using a partition decorator (table$YYYYMMDD) with WRITE_TRUNCATE. Each partition is replaced atomically, the load never rescans history, and reruns are safe because every load truncates only its own partition. The pattern covers late-arriving data while keeping scan cost proportional to the delta, not the table size.
refresh_window task (io.kestra.plugin.core.flow.Loop) iterates over the day_offsets input with concurrencyLimit: 1, processing one day at a time so partition loads never collide.load_partition (io.kestra.plugin.gcp.bigquery.LoadFromGcs) reads gs://.../events/dt=YYYY-MM-DD/*.parquet for that day and loads it into analytics.events$YYYYMMDD with format: PARQUET and writeDisposition: WRITE_TRUNCATE. The date is computed from now() and the offset via dateAdd.allowFailure: true on the load lets a day with no matching files pass without failing the run.notify task (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts a Slack confirmation summarizing how many days were refreshed.daily Schedule trigger (cron: 0 5 * * *) is ready to enable, and an alert_on_failure error handler posts to Slack if the refresh fails.BigQuery scheduled queries can run SQL on a cron, but they cannot orchestrate a per-day GCS-to-partition load loop, react to upstream events, retry a single failed day, or fan a result into Slack alerting and lineage. Kestra adds event and schedule triggers, per-task retries and error handlers, full execution lineage, and a declarative YAML definition you version in Git. You control exactly which partitions are rewritten and when, instead of rescanning the whole table on a fixed schedule.
A day-partitioned BigQuery table (analytics.events) and GCS objects laid out under events/dt=YYYY-MM-DD/ as Parquet.
GCP_PROJECT_ID: GCP project hosting the dataset.GCP_SERVICE_ACCOUNT: service account key JSON with BigQuery job and GCS read access.GCS_BUCKET: bucket holding the day-partitioned source files.SLACK_WEBHOOK_URL: Slack incoming webhook URL for notifications and alerts.from path and destinationTable at your partitioned data.day_offsets to cover how far back late data can arrive (for example [0,-1,-2,-3,-4,-5,-6] for a week).daily trigger.day_offsets to trade freshness against cost.format: PARQUET for CSV or AVRO to match your exports.refresh_window.concurrencyLimit if your partitions are independent and you want faster backfills.