Webhook icon
Schedule icon
WorkingDirectory icon
Commands icon
Docker icon
Script icon
If icon
SlackIncomingWebhook icon
Fail icon
Switch icon
Log icon

Gate KEDA Scaling Health on a Burned Error Budget

Score KEDA ScaledObject trigger failures and HPA replica flaps against a burn error budget, alert Slack when it is exhausted, and fail the gate.

Categories
Infrastructure

How it works

  1. on_status_publish (io.kestra.plugin.core.trigger.Webhook, keyed with the KEDA_STATUS_WEBHOOK_KEY secret) receives the ScaledObject and HPA status JSON from your KEDA status exporter or a CI job. daily_budget_rollup (io.kestra.plugin.core.trigger.Schedule, cron 0 6 * * *) is the safety net and ships disabled: true until an operator turns it on.
  2. status_workspace (io.kestra.plugin.core.flow.WorkingDirectory) opens one shared directory so the staged fixture and the scorer see the same file.
  3. write_status_fixture (io.kestra.plugin.scripts.shell.Commands on the io.kestra.plugin.scripts.runner.docker.Docker runner, image alpine:3.20) writes status.json with a heredoc. The fixture holds one failing queue trigger with ten failures and an HPA that flapped six times across twenty two scale events, a realistic thrash window. entryPoint: [] clears the image entrypoint so the shell script runs instead of the default command.
  4. score_error_budget (io.kestra.plugin.scripts.python.Script on python:3.12-slim) prefers the posted webhook body when present and falls back to the staged file. It burns five budget points per trigger failure and four per replica flap against a hundred point budget, caps the total at one hundred, and calls the state exhausted when the burned share meets exhausted_threshold_pct. The state, the consumed share, the per trigger counters, the flap count and the worst trigger are printed as a single ::{"outputs": ...}:: line.
  5. burn_gate (io.kestra.plugin.core.flow.If) branches on budget_state == 'exhausted'. An exhausted budget reaches alert_exhausted (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) and then fail_gate (io.kestra.plugin.core.execution.Fail). Every other state reaches route_remaining (io.kestra.plugin.core.flow.Switch), whose map shaped cases record a within budget burn on log_within_budget and fall through to log_unknown_budget when the status could not be parsed.
  6. The flow level errors handler (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) reports any failure with the execution id, because a dead scorer looks exactly like a healthy scaler.

What you get

  • A numeric error budget that turns vague scaler flapping into a burn rate you can threshold and trend.
  • A hard gate for deploy pipelines, an exhausted budget fails the execution so nothing ships on top of a thrashing scaler.
  • Channel context that names the worst trigger and the exact counters instead of a bare alert.
  • budget_state, budget_consumed_pct, failed_trigger_count, total_trigger_failures, replica_flaps and worst_trigger as typed outputs for dashboards and weekly reliability reviews.
  • A disabled daily schedule as a second net once the webhook path is trusted.

Who it's for

  • Platform teams running KEDA autoscaling who need a gate between scaler health and production deploys.
  • SREs on call for HPA thrash who want the failing trigger named before the page fires.
  • Kubernetes operators tuning trigger error budgets who need burned share trends per ScaledObject.

Why orchestrate this with Kestra

KEDA reports trigger health per ScaledObject but nothing scores that health over time or stops a deploy when the error budget is gone. Kestra gives the check a keyed webhook for on demand scoring, a disabled schedule as the safety net, a fixture driven scorer whose burn model lives in version control, an If that isolates the exhausted state, a Switch that separates a within budget burn from an unparseable status, and routing into Slack and execution failure from one declarative file. A cron job with a curl script gives you none of the history, the branching or the gateable outputs.

Prerequisites

  • A worker with Docker access, the fixture runs in alpine:3.20 and the scorer in python:3.12-slim.
  • A KEDA status exporter or CI job that can POST ScaledObject and HPA status JSON to the webhook, or rely on the staged fixture for trial runs.
  • A Slack incoming webhook for the exhausted alert and for failure alerts.

Secrets

  • KEDA_STATUS_WEBHOOK_KEY: key segment guarding the status webhook URL, anyone holding it can trigger a scoring run.
  • SLACK_WEBHOOK_URL: Slack incoming webhook for the exhausted alert and failure alerts.

Quick start

  1. Add the two secrets to your Kestra namespace.
  2. Run the flow once with the default inputs and confirm budget_state reads exhausted, budget_consumed_pct reads 74.0, total_trigger_failures reads 10, replica_flaps reads 6 and worst_trigger reads checkout-sqs.
  3. Re-run with exhausted_threshold_pct at 90 to watch the same burn route through the Switch within case to log_within_budget instead of failing the gate.
  4. Point your status exporter at /api/v1/{tenant}/executions/webhook/{namespace}/keda-scaledobject-error-budget-gate/{KEDA_STATUS_WEBHOOK_KEY} and enable daily_budget_rollup once the path is trusted.

How to extend

  • Weight the burn model differently per trigger type by extending the scoring loop in score_error_budget, for example charging more points for queue depth triggers than for cron triggers.
  • Push the scored window into a time series database with io.kestra.plugin.jdbc.postgresql.Queries to chart budget burn per ScaledObject over weeks.
  • Fan out across many ScaledObjects with io.kestra.plugin.core.flow.Loop over a subflow that scores one status document each.
  • Swap the Slack exhausted alert for a PagerDuty or Opsgenie integration when the budget burn should page instead of post.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.