Schedule icon
Request icon
Switch icon
Log icon
SlackIncomingWebhook icon

Pyrra Service Level Objective (SLO) Error Budget Sentinel

Monitor Pyrra SLO compliance, track error budget burn rates, detect service level degradation, and dispatch automated Slack incident alerts with Kestra.

Categories
Infrastructure

Continuously monitors Pyrra Service Level Objective (SLO) tracking services, inspects multi-window error budget burn rates for critical tier-1 microservices, and dispatches actionable Slack incident notifications upon error budget exhaustion or API degradation.

How it works

  1. A recurrent Schedule trigger initiates the sentinel audit workflow every hour (disabled by default for on-demand configuration).
  2. The probe_pyrra_readiness task executes a GET request against Pyrra's /readyz endpoint to confirm controller readiness and Prometheus connectivity.
  3. The probe_active_objectives task queries /api/v1/objectives to verify active SLO CRD synchronization and objective definitions.
  4. The probe_service_slo_status task calls /api/v1/objectives/status to evaluate real-time multi-window error budget burn rates and availability status for target microservices.
  5. The evaluate_slo_compliance task uses Pebble conditionals inside a Switch task to categorize outcomes into COMPLIANT, ENGINE_UNHEALTHY, OBJECTIVES_UNAVAILABLE, or SLO_STATUS_ERROR.
  6. Upon detecting outages or status retrieval degradation, contextual Slack notifications are dispatched via io.kestra.plugin.slack.notifications.SlackIncomingWebhook with diagnostic status codes, and an audit trail is recorded in Kestra's execution history.

What you get

  • Automated operational surveillance over Pyrra SLO controllers and Prometheus metric ingestion pipelines.
  • Proactive alerting before error budgets are completely depleted, safeguarding microservice availability agreements.
  • Contextual incident routing to dedicated SRE and platform engineering Slack channels.
  • Automated deployment gate capability to halt downstream CD pipelines when error budgets are exhausted.
  • Centralized, revision-controlled audit history for regulatory and post-incident compliance reviews.

Who it's for

  • Site Reliability Engineers (SREs) managing production Service Level Objectives and error budgets.
  • Platform Engineers responsible for observability infrastructure, Prometheus, and Pyrra controllers.
  • DevOps Teams operating automated deployment gates dependent on error budget availability.

Why orchestrate this with Kestra

  • Declarative Reliability Guardrails: Maintain SLO audit schedules, target services, and incident runbooks as version-controlled YAML blueprints.
  • Native Credential Isolation: Inject Slack webhook credentials via namespace secrets using {{ secret(...) }} without plaintext exposure.
  • Multi-Branch Flow Routing: Differentiate between controller outages, API parsing failures, and service-level burn rate spikes via native Switch tasks.
  • Auditable Execution Telemetry: Maintain complete execution logs and HTTP diagnostic status codes across every audit run.

Prerequisites

  • A running Pyrra instance (accessible over HTTP) with standard health and API endpoints exposed (default port 9099).
  • Network reachability from the Kestra worker or runner environment to the Pyrra service.
  • A Slack Incoming Webhook URL configured with permission to post into your target operations channel.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used for dispatching reliability alerts and incident notifications.

Quick start

  1. Configure the SLACK_WEBHOOK_URL secret in your target Kestra namespace settings.
  2. Import this blueprint into your Kestra instance (e.g., under namespace company.sre).
  3. Click Execute with default parameters or supply your custom pyrra_api_endpoint and target_service_name.
  4. Confirm that execution logs display successful HTTP response codes under the Executions tab.
  5. Activate the scheduled_sentinel_audit trigger to establish recurrent automated SLO surveillance.

How to extend

  • Automated Deployment Gate: Chain this flow as a prerequisite task in CI/CD pipelines to block Canary rollouts whenever error budget burn exceeds thresholds.
  • Multi-Service Sweep: Wrap the objective status probe inside a Loop task to evaluate error budgets across all frontend, payment, and inventory services in a single workflow.
  • PagerDuty Escalation: Route ENGINE_UNHEALTHY events directly to PagerDuty or Opsgenie on-call escalation schedules.
  • Automated Scaledown: Trigger a Kubernetes Horizontal Pod Autoscaler (HPA) adjustment or traffic throttling workflow upon sustained high error budget burn.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.