Snapshot Loki Logs Automatically When an Incident Starts

The first ten minutes of an incident are usually spent assembling context that already exists. This blueprint attaches that context automatically: the incident tooling calls a Kestra webhook, io.kestra.plugin.grafana.loki.QueryRange captures the last 15 minutes of the affected service's logs, and Slack gets a summary with the line count while the full snapshot stays on the execution. By the time responders join the channel, the window in which things went wrong is already preserved, immune to log retention and later noise.

How it works

1. The incident_webhook trigger (io.kestra.plugin.core.trigger.Webhook) exposes a URL your incident tooling calls at declaration time; the service input defaults to checkout and can be overridden per call.
2. snapshot_logs (io.kestra.plugin.grafana.loki.QueryRange) runs {app="<service>"} with since: 15m, fetching up to 2000 lines from the window leading into the incident.
3. The lines land in {{ outputs.snapshot_logs.logs }} on the execution, a frozen record of the pre-incident window.
4. post_summary posts the scalar line count and service name to Slack, pointing responders at the execution for the full set.
5. The errors block warns the channel explicitly when the snapshot fails, so nobody waits on context that is not coming.

What you get

• Log context captured at the moment of declaration, before retention, rate limits, or recovery noise erode it.
• A Slack summary that tells responders what exists and where, without pasting thousands of lines into the channel.
• A per-incident execution record that doubles as postmortem evidence.
• A webhook contract simple enough to wire into PagerDuty, Opsgenie, or a plain curl in a runbook.

Who it's for

• On-call engineers tired of reconstructing the timeline from memory and expired logs.
• Incident commanders who want context capture to be a zero-step part of declaration.
• SRE teams building automated incident enrichment on top of existing Loki investment.

Why orchestrate this with Kestra

The query is trivial; the guarantees are not. Kestra gives the snapshot a stable webhook endpoint, an input contract for the affected service, retries against a briefly unreachable Loki, a permanent execution holding the captured lines, and an explicit failure signal in the incident channel. Extending the same flow to capture metrics, recent deploys, or config diffs is adding tasks, not building a new service.

Prerequisites

• Incident tooling able to call a webhook URL at declaration (or an engineer with curl).
• A reachable Loki instance with services labeled by app (adjust the selector to your labels).
• A Slack incoming webhook posting into the incident channel.
• For authenticated or multi-tenant Loki, the task also accepts authToken and tenantId (X-Scope-OrgID).

Secrets

• LOKI_URL: Loki base URL, e.g. http://loki.monitoring:3100.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the LOKI_URL and SLACK_WEBHOOK_URL secrets to your Kestra namespace.
2. Replace change-me-strong-webhook-key with a strong random value.
3. Call the webhook URL with a test service value and confirm the Slack summary arrives.
4. Wire the URL into your incident tooling's declaration hook.

How to extend

• Pass the service name from the webhook body instead of the input default, so one endpoint serves every service.
• Widen the window with since: 60m for slow-burn incidents, raising limit to match.
• Add a second QueryRange filtered to |= "ERROR" and report both totals in the summary.
• Ship the snapshot to object storage with a downstream task when postmortems need the lines outside Kestra.

Links

• Grafana Loki plugin documentation
• LogQL documentation