Compare Error Logs Between Releases with Loki

Every deploy answers one question badly: is the new version actually worse. Dashboards show the total, not the split. This blueprint answers it directly with two io.kestra.plugin.grafana.loki.QueryRange tasks that count ERROR lines per version label over the same one hour window, one for the trusted baseline and one for the fresh release. If the new version out-errors the old one, Discord gets both counts and a rollback suggestion; otherwise the flow records a pass.

How it works

1. old_version_errors and new_version_errors (io.kestra.plugin.grafana.loki.QueryRange) each run {app="checkout", version="..."} |= "ERROR" with since: 1h, so both releases are measured over the identical window.
2. The version labels come from the old_version and new_version inputs, defaulting to v1.4.2 and v1.5.0, so the flow runs manually or from a deploy pipeline passing real values.
3. compare_versions (io.kestra.plugin.core.flow.If) compares the two scalar counts with {{ (outputs.new_version_errors.logs | length) > (outputs.old_version_errors.logs | length) }}.
4. On a regression, suggest_rollback posts both counts and the version names to Discord; on a pass, log_release_healthy records the same numbers in the execution log.
5. The errors block alerts when the comparison itself fails, because a missing verdict is easy to misread as a green one.

What you get

• A per-release error comparison that dashboards aggregating by service cannot show.
• A concrete rollback signal minutes after deploy, with both numbers in the message.
• Version labels driven by inputs, so a CI pipeline can call the flow with the exact tags it just shipped.
• The matched lines from both versions kept on the execution for immediate diagnosis when the check trips.

Who it's for

• Teams running canary or rolling deploys where old and new versions log side by side.
• Release engineers who want an automated post-deploy verdict instead of ten minutes of dashboard staring.
• Platform teams wiring quality gates into pipelines without adding a metrics stack requirement.

Why orchestrate this with Kestra

The comparison is two queries and an if statement, but a release gate needs to run at the right moment, with the right version values, produce a verdict somewhere visible, and never fail silently. Kestra provides the input contract for the pipeline, the branching, the notification, and a per-deploy execution history that shows every verdict this gate has ever produced.

Prerequisites

• Logs labeled with both app and version in Loki, so releases are distinguishable by selector.
• A window where both versions run side by side (canary or rolling deploys), or accept that the old version's count drops to zero after cutover.
• A Discord incoming webhook for the rollback suggestions and failure alerts.
• For authenticated or multi-tenant Loki, the tasks also accept authToken and tenantId (X-Scope-OrgID).

Secrets

• LOKI_URL: Loki base URL, e.g. http://loki.monitoring:3100.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the LOKI_URL and DISCORD_WEBHOOK_URL secrets to your Kestra namespace.
2. Adjust the label selector to your app and version labeling convention.
3. Execute the flow with two version values that exist in your logs and check the verdict.
4. Call the flow from your deploy pipeline with the real old and new tags after each release.

How to extend

• Trigger the flow from a deploy webhook by adding io.kestra.plugin.core.trigger.Webhook and mapping the body to the inputs.
• Add a tolerance to the condition, for example requiring the new count to exceed the old one by 20 percent before alerting.
• Chain an automated rollback task in the regression branch once the team trusts the signal.
• Compare WARN lines or a specific error code by adjusting the match string in both queries.

Links

• Grafana Loki plugin documentation
• LogQL documentation