GCP Firestore Stuck Document Status Sentinel

A write-retry pipeline, a Cloud Function trigger, or a client-side offline sync can all leave a Firestore document stuck on a failed status forever - Firestore itself has no concept of "this document has been stuck for too long" and raises no alert on its own. This blueprint queries a collection for documents matching a specific stuck-status value and alerts Slack with the count the moment any exist.

How it works

1. periodic_stuck_document_check (io.kestra.plugin.core.trigger.Schedule) runs every 15 minutes, explicitly passing gcp_project_id, firestore_collection, status_field, stuck_status_value, and max_documents_to_check through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
2. probe_stuck_documents (io.kestra.plugin.gcp.firestore.Query) runs an EQUAL_TO filter on status_field, fetching up to max_documents_to_check matching documents and returning size.
3. evaluate_stuck_documents (io.kestra.plugin.core.flow.If) branches on whether size > 0.
   • then (NEEDS_ATTENTION): alert_stuck_documents reports the count to Slack.
   • else (HEALTHY): log_collection_healthy records a clean check.
4. log_audit_complete always runs last, printing the stuck count regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable database or bad credentials must not read as HEALTHY.

What you get

• A HEALTHY/NEEDS_ATTENTION answer sourced directly from a live Firestore query, not from a separate logging pipeline that has to be built and maintained alongside the application.
• A bounded query (limit) so a large stuck-document backlog cannot turn a routine health check into an expensive, unbounded read.
• A worked example of io.kestra.plugin.gcp.firestore.Query's EQUAL_TO filter and FETCH mode, documented here specifically because the task's inequality operators cannot be used safely against a native Timestamp field - see Pitfalls.

Who it's for

• Teams running Firestore-backed sync, queueing, or write-retry pipelines who need an early signal when documents start piling up in a failed state.
• On-call engineers who want one Slack alert with the exact stuck count instead of opening the Firestore console to run the same query by hand.
• Anyone evaluating the GCP Firestore plugin who wants a worked example of Query used as a governance probe rather than a data-fetch step in an ETL pipeline.

Why orchestrate this with Kestra

Firestore itself has no scheduler and no "alert me if documents match this filter" primitive - checking requires either a Cloud Function wired to every write (more code to maintain) or someone remembering to run the query by hand. Kestra supplies the schedule, the HEALTHY/NEEDS_ATTENTION classification as a first-class branch, and an execution history that shows exactly when a backlog started and how large it grew.

Prerequisites

• A GCP project with a Firestore database containing the collection to audit (update gcp_project_id, firestore_collection).
• A GCP service account JSON key with Firestore read access (roles/datastore.viewer or broader).
• A Slack incoming webhook for alerts.

This requires a real GCP project for production use. For fully local, account-free testing, Google ships an official Firestore emulator as part of the Firebase CLI / Cloud SDK (a different product from third-party emulator services, with no account or billing requirement for pure local use) - see Pitfalls for the one operational catch that applies to using it with this specific task.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: service account key used by io.kestra.plugin.gcp.firestore.Query.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_stuck_documents and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_SLACK_WEBHOOK_URL=$(echo -n '<your-webhook-url>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

Input	Type	Default	Allowed values
gcp_project_id	STRING	<your-gcp-project-id>	Valid GCP project ID
firestore_collection	STRING	sync_jobs	Any Firestore collection name
status_field	STRING	status	Any document field name
stuck_status_value	STRING	FAILED	Any status string value
max_documents_to_check	INT	100	Any positive integer

Outputs

• {{ outputs.probe_stuck_documents.size }}: number of matching documents found this run.
• {{ outputs.probe_stuck_documents.rows }}: list of matching raw document maps (populated when fetchType: FETCH is used).

Quick start

1. Create a Firestore database and a sync_jobs collection with at least one document.
2. Add the GCP_SERVICE_ACCOUNT_JSON and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually against documents with status values other than FAILED and confirm log_collection_healthy fires.
4. Set one document's status field to FAILED and re-run to confirm alert_stuck_documents fires with a count of at least 1.
5. Enable periodic_stuck_document_check once you trust the check against your real collection.

How to extend

• Loop over several firestore_collection / stuck_status_value combinations with io.kestra.plugin.core.flow.Loop to audit multiple pipelines' stuck-document states in one run.
• Surface a few of the actual stuck document IDs (from rows) in the Slack payload instead of only the count, once fetchType: FETCH_ONE or a small limit makes that practical for the channel.
• Add a second, separate remediation flow that re-queues or deletes confirmed stuck documents behind an explicit approval gate - deliberately kept out of this sentinel so the monitoring path never modifies data.

Pitfalls

• io.kestra.plugin.gcp.firestore.Query's Filter.value only supports strings (verified from the task's own Filter class, which types value as Property<String> with the schema note "Only strings are supported at the moment"). This is why this blueprint uses EQUAL_TO on a string status field, and specifically does not attempt a LESS_THAN/GREATER_THAN range filter against a native Firestore Timestamp field (such as a TTL-policy field) - that comparison would not produce a correct result against this task as currently implemented.
• Firestore's own documented TTL policy deletion SLA is "typically within 24 hours, not guaranteed" (per Firebase's own TTL documentation) - a related but different governance question from this blueprint's stuck-status check, and one this specific plugin cannot currently answer safely for the reason above.
• io.kestra.plugin.gcp.firestore.Query has no explicit "point at a local emulator" property, the same situation as the GCP Pub/Sub plugin. The underlying Google Firestore Java client does honor the official FIRESTORE_EMULATOR_HOST environment variable, but that variable has to be set on the Kestra worker process itself, not anywhere inside this flow's YAML.
• serviceAccount must still be a syntactically valid service-account JSON even against an emulator, since credential resolution is shared GCP-plugin code that parses it before any network call happens - a well-formed placeholder key works for emulator testing, but an empty or malformed value fails before the request is sent.
• This blueprint is UNTESTED against a live GCP project or the Firestore emulator. The Query task's Filter behavior and FetchOutput fields are taken from the plugin's own source on main, and the TTL deletion SLA caveat is taken from Firebase's own TTL documentation - but no Kestra engine was run to execute this flow end to end.

Links

• Kestra GCP plugin documentation
• Firestore TTL policy documentation
• Firestore emulator documentation
• Kestra secrets (OSS vs Enterprise)