Canary Deployment Check Backed by Prometheus

A canary deployment without an automated verdict is just a slow rollout. This blueprint makes the verdict a flow. The deploy pipeline calls a Kestra webhook the moment the canary starts taking traffic. Ten minutes of signal later, io.kestra.plugin.prometheus.Query computes the canary's 5xx error rate with fetchType: FETCH_ONE, and an If task delivers the verdict. Healthy, the flow logs the measured rate and exits green, telling the pipeline to promote. Unhealthy, it pages Discord with the number and fails through io.kestra.plugin.core.execution.Fail, stopping the rollout.

How it works

1. deploy_webhook (io.kestra.plugin.core.trigger.Webhook) starts the flow when the deploy pipeline calls the webhook URL with the configured key. Replace change-me-strong-webhook-key with a strong random value before use.
2. canary_error_rate (io.kestra.plugin.prometheus.Query, fetchType: FETCH_ONE) evaluates the PromQL ratio scoped to version="canary" over a 10 minute window, exposing the single row as {{ outputs.canary_error_rate.row }}.
3. verdict (io.kestra.plugin.core.flow.If) compares the scalar against a 1 percent threshold. The healthy branch logs the rate; the unhealthy branch pages Discord, then fails the execution with the rate in the errorMessage.
4. The errors block posts a Discord alert on any failed execution, covering both a burning canary and a broken query, so a rollout never proceeds on missing data.

What you get

• A canary verdict as a real execution state that deploy tooling can wait on.
• The measured error rate in the log, the page, and the failure message, so rollback decisions start from a number.
• A meta-alert when the check itself breaks, closing the gap where a broken query looks like a passing canary.
• Version-controlled verification logic that changes by pull request, not by editing CI scripts.

Who it's for

• Platform teams adding automated verification to an existing canary or blue-green deploy process.
• SRE teams who want rollback triggered by metrics rather than by whoever is watching the dashboard.
• Delivery teams making promote-or-rollback decisions auditable.

Why orchestrate this with Kestra

CI systems are good at shipping artifacts and bad at waiting on metrics. Moving the verification into Kestra gives the check retries, a full execution history of every verdict with the measured rate, secrets management for the Prometheus endpoint, and a webhook interface any pipeline can call with one curl command.

Prerequisites

• A Prometheus server scraping the service with a version label distinguishing the canary, or adjust the PromQL selector to your labels.
• Traffic flowing to the canary during the window; with zero canary requests the ratio is undefined and the query returns no row, which fails the check rather than passing it.
• A deploy pipeline able to call the webhook after the canary goes live.
• A Discord incoming webhook for pages.
• The Query task supports HTTP basic auth through its username and password properties if Prometheus sits behind one.

Secrets

• PROMETHEUS_URL: base URL of the Prometheus server, e.g. http://prometheus:9090.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the PROMETHEUS_URL and DISCORD_WEBHOOK_URL secrets to your Kestra namespace.
2. Replace the webhook key with a strong random value.
3. Adjust the PromQL selector to how your deploys label the canary version.
4. Call the webhook from your deploy pipeline after the canary receives traffic, and gate promotion on the execution finishing in Success state.

How to extend

• Pass the version under test in the webhook body and use it in the query selector through {{ trigger.body }}.
• Add latency to the verdict with a second Query on a p99 histogram quantile, and require both to pass.
• Trigger an automated rollback task in the unhealthy branch, such as a Kubernetes rollout undo, before failing.
• Wait longer for signal by adding a io.kestra.plugin.core.flow.Pause task between the webhook and the query.

Links

• Prometheus plugin documentation
• PromQL documentation