GCP Bigtable Row Write Freshness Sentinel

A device, sensor, or ingestion pipeline that writes continuously into a single Bigtable row can stop silently - Bigtable itself has no concept of "this row should be updated every few minutes and has not been." This blueprint reads the single most recent cell version under a row key prefix and alerts the moment it falls behind an acceptable staleness window, or is missing outright.

How it works

1. periodic_freshness_check (io.kestra.plugin.core.trigger.Schedule) runs every 15 minutes, explicitly passing gcp_project_id, bigtable_instance_id, bigtable_table_id, row_key_prefix, max_staleness_minutes, and bigtable_emulator_host through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
2. probe_latest_row (io.kestra.plugin.gcp.bigtable.ReadRows, fetchType: FETCH_ONE) reads the single row matching row_key_prefix, returning row.cells[0] as the most recently written cell - Bigtable's own documented behavior returns cell versions newest-first within a row.
3. evaluate_row_presence (io.kestra.plugin.core.flow.If) branches on whether any row matched at all.
   • else (NO_MATCHING_ROW): alert_no_matching_row reports that the prefix matched nothing.
   • then (row found): nested evaluate_staleness (io.kestra.plugin.core.flow.If) branches on whether the latest cell's timestamp is older than max_staleness_minutes.
     • then (STALE): alert_stale_row reports the last-write time to Slack.
     • else (FRESH): log_row_fresh records a clean check.
4. log_audit_complete always runs last, printing whether a row was found regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable Bigtable instance or bad credentials must not read as any of the three handled states.

What you get

• A FRESH/STALE/NO_MATCHING_ROW answer sourced directly from the row's own most recent write, not from a separate change-data-capture pipeline that has to be built and kept in sync.
• A distinct NO_MATCHING_ROW state for a writer that never started or a row that was deleted, which a staleness-only check would silently miss.
• A flow that is genuinely runnable fully locally, with zero GCP account or credentials, via emulatorHost - verified from the plugin's own source to use NoCredentialsProvider when set, unlike this repository's GCP Pub/Sub and Firestore blueprints, which both need their emulator variable set on the Kestra worker process itself rather than inside the flow.
• A worked example of io.kestra.plugin.gcp.bigtable.ReadRows used for a single-row freshness probe rather than a bulk data export.

Who it's for

• IoT and telemetry platform teams writing device or sensor readings into Bigtable who need an early signal when a device stops reporting.
• On-call engineers who want one Slack alert with the exact last-write time instead of querying Bigtable by hand to check whether a feed is still live.
• Anyone evaluating the GCP Bigtable plugin who wants a worked example of ReadRows's emulatorHost property, since every example in the plugin's own documentation targets real Bigtable.

Why orchestrate this with Kestra

Bigtable itself has no scheduler and no "alert me if this row goes stale" primitive - checking requires either instrumenting every writer with its own heartbeat logic or someone remembering to query the row by hand. Kestra supplies the schedule, the FRESH/STALE/NO_MATCHING_ROW classification as a first-class branch, and an execution history that shows exactly when a writer stalled and for how long.

Prerequisites

• A Bigtable instance and table with a row written to continuously under a known row key prefix (update bigtable_instance_id, bigtable_table_id, row_key_prefix), or a local emulator for testing.
• A GCP service account JSON key with roles/bigtable.reader - not needed at all when using bigtable_emulator_host.
• A Slack incoming webhook for alerts.

Local testing (official Bigtable emulator, no GCP account needed):

docker run -d --name bigtable-emulator -p 8086:8086 google/cloud-sdk gcloud beta emulators bigtable start --host-port=0.0.0.0:8086
docker network connect YOUR_KESTRA_NETWORK bigtable-emulator
cbt -project emulator-test -instance my-instance createtable sensor-readings
cbt -project emulator-test -instance my-instance createfamily sensor-readings cf1
cbt -project emulator-test -instance my-instance set sensor-readings "device-42#001" cf1:reading=72

Run docker network ls first to find the active Kestra network name; cbt is the Bigtable CLI bundled with the Cloud SDK, used here against the emulator with BIGTABLE_EMULATOR_HOST=localhost:8086 exported first. From inside the Kestra worker's Docker network, set bigtable_emulator_host to bigtable-emulator:8086 instead of localhost:8086.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: service account key used by io.kestra.plugin.gcp.bigtable.ReadRows against real Bigtable. Can be left as any placeholder value when bigtable_emulator_host is set, since the emulator path never reads it.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_stale_row, alert_no_matching_row, and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_SLACK_WEBHOOK_URL=$(echo -n '<your-webhook-url>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

Input	Type	Default	Allowed values
gcp_project_id	STRING	<your-gcp-project-id>	Valid GCP project ID (ignored when emulatorHost is set)
bigtable_instance_id	STRING	my-instance	Bigtable instance ID
bigtable_table_id	STRING	sensor-readings	Bigtable table ID
row_key_prefix	STRING	device-42#	Row key prefix resolving to a single row
max_staleness_minutes	INT	30	Any positive integer
bigtable_emulator_host	STRING	""	host:port for local emulator testing, or empty for production

Outputs

• {{ outputs.probe_latest_row.row }}: the matched row map (rowKey and cells list with microsecond timestamp); null when no row matched.
• {{ outputs.probe_latest_row.rowCount }}: 1 when a row was found, 0 otherwise.

Quick start

1. Start the Bigtable emulator and create a test row with the commands above.
2. Add the SLACK_WEBHOOK_URL secret (GCP_SERVICE_ACCOUNT_JSON can stay a placeholder).
3. Run the flow manually with bigtable_emulator_host: "localhost:8086" against the just-written row and confirm log_row_fresh fires.
4. Wait past max_staleness_minutes (or lower it for the test) without writing again, then re-run to confirm alert_stale_row fires.
5. Point row_key_prefix at a prefix with no matching row and re-run to confirm alert_no_matching_row fires.
6. Clear bigtable_emulator_host and point this at a real Bigtable instance once you trust the check; enable periodic_freshness_check once you trust the cadence.

How to extend

• Loop over several row_key_prefix values with io.kestra.plugin.core.flow.Loop to freshness-check every monitored device or entity in one run.
• Read cells[1], cells[2], and so on (older versions of the same cell, still newest-first) to additionally detect a writer that is updating on a slower cadence than normal, not just one that stopped outright.
• Add a columnFamily filter if the table carries multiple column families and only one represents the writer's own heartbeat data.

Pitfalls

• row_key_prefix must narrow to exactly one row for this freshness check to mean anything. fetchType: FETCH_ONE returns the row with the lexicographically smallest key matching the prefix, not the most recently written row across multiple matches - if the prefix matches several rows, this blueprint silently checks only one of them.
• Bigtable cell timestamps are microseconds since the Unix epoch, not milliseconds (verified from Cloud Bigtable's own client library documentation) - evaluate_staleness divides by 1,000,000 before converting to a date for exactly that reason; dividing by 1,000 as if it were milliseconds would make every row look roughly 1,000x more stale than it actually is.
• emulatorHost is explicitly documented as "not intended for production use" by the plugin itself - this blueprint's bigtable_emulator_host input must be cleared before pointing the flow at a real account.
• A row that exists but has zero cells matching no column-family filter is not explicitly guarded. Such a row would make row.cells[0] throw while rendering, which surfaces through the errors block rather than a silent misclassification.
• This blueprint is UNTESTED against a live Bigtable instance or the emulator. The ReadRows properties and output shape, the emulatorHost no-credentials behavior, and the microsecond timestamp unit are all taken from the plugin's own source on main and Cloud Bigtable's own client library documentation - but no Docker container or Kestra engine was run to execute this flow end to end.

Links

• Kestra GCP plugin documentation
• Cloud Bigtable overview documentation
• Cloud Bigtable emulator documentation
• Kestra secrets (OSS vs Enterprise)