Detect Silent Services with a Loki Heartbeat Check

Error alerting has a blind spot: a service that crashes hard, loses its log shipper, or hangs entirely produces no errors at all. The absence of logs is the signal. This blueprint inverts the usual pattern with io.kestra.plugin.grafana.loki.QueryRange: every hour it fetches the payments service heartbeat lines from the last hour, and if the query returns zero lines the flow alerts Slack and deliberately fails, turning silence into a red execution nobody can miss.

How it works

1. check_heartbeat (io.kestra.plugin.grafana.loki.QueryRange) runs {app="payments"} |= "heartbeat" with since: 1h, so the window always covers the hour before the check.
2. evaluate_silence (io.kestra.plugin.core.flow.If) branches on the scalar count {{ (outputs.check_heartbeat.logs | length) == 0 }}.
3. On silence, alert_silence posts a Slack alert and fail_execution (io.kestra.plugin.core.execution.Fail) marks the run failed, so the outage is visible in the execution history and to anything monitoring flow states.
4. On a healthy hour, log_healthy records the heartbeat count and the run ends green.
5. The errors block sends a distinct alert when the check itself breaks; note that the deliberate Fail also routes through it, which is intentional, silence should page twice rather than zero times.
6. A disabled-by-default Schedule trigger runs the check hourly.

What you get

• Detection of the failure mode error alerting cannot see: a service that stopped logging entirely.
• A hard execution failure on silence, so flow-state dashboards and SLA monitors pick it up without custom wiring.
• A green-run audit trail proving the service was logging every hour it was checked.
• A pattern that works with any regularly emitted line, heartbeats, cron job completions, or consumer poll logs.

Who it's for

• SRE teams closing the gap between "no alerts" and "actually healthy".
• Platform engineers who have seen a dead log shipper mask a real outage.
• Teams running batch consumers or daemons whose only health signal is their own log output.

Why orchestrate this with Kestra

Loki can tell you a query returned nothing, but acting on absence needs a scheduler that runs reliably, branching logic, an alert channel, and a way to make the silent hour permanently visible. Kestra provides the schedule, the If branch, the deliberate Fail, and the execution history in one declarative flow, and the check itself failing raises its own alarm.

Prerequisites

• A service that logs a predictable heartbeat line at least hourly, or any regular log statement to match on.
• A reachable Loki instance receiving those logs.
• A Slack incoming webhook for alerts.
• For authenticated or multi-tenant Loki, the task also accepts authToken and tenantId (X-Scope-OrgID).

Secrets

• LOKI_URL: Loki base URL, e.g. http://loki.monitoring:3100.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the LOKI_URL and SLACK_WEBHOOK_URL secrets to your Kestra namespace.
2. Point the LogQL query at a line your service emits reliably within the window.
3. Execute the flow once while the service is up and confirm the healthy branch logs a count.
4. Set disabled: false on the hourly trigger.

How to extend

• Tighten the window with since: 15m and a more frequent cron for services with strict availability targets.
• Duplicate the flow per service, or drive the label selector from a flow input and iterate with a ForEach.
• Require a minimum count instead of nonzero by changing the If condition, useful for heartbeats with a known cadence.
• Add a remediation task in the silence branch, for example restarting the service before the alert goes out.

Links

• Grafana Loki plugin documentation
• LogQL documentation