Schedule icon
Request icon
Switch icon
SlackIncomingWebhook icon
Log icon

Grafana Pyroscope Ingestion Health & Application Coverage Sentinel

Audit Grafana Pyroscope ingester readiness, track continuous profiling app coverage, and dispatch Slack alerts on profiling dropouts.

Categories
Infrastructure

Monitors standalone and cluster Grafana Pyroscope continuous profiling deployments in real time, verifying ingester readiness and asserting that expected microservice applications are actively streaming profiling data to prevent blind spots during performance investigations.

How it works

  1. periodic_profiling_audit (io.kestra.plugin.core.trigger.Schedule) fires every 15 minutes with target connection details and coverage thresholds. Shipped disabled by default to facilitate safe initial verification.
  2. probe_pyroscope_readiness (io.kestra.plugin.core.http.Request) probes GET /ready to ensure the Pyroscope ingester ring and tenant distributors are fully operational.
  3. probe_active_apps (io.kestra.plugin.core.http.Request) queries GET /api/apps to retrieve the list of all currently reporting profiling applications.
  4. classify_pyroscope_status (io.kestra.plugin.core.flow.Switch) routes execution:
    • INGESTER_DEGRADED: Triggers when /ready returns a non-200 status code (e.g. 503 during ingester ring backpressure or storage saturation), dispatching an urgent alert to #observability-alerts via io.kestra.plugin.slack.notifications.SlackIncomingWebhook.
    • APPLICATION_COVERAGE_DEFICIT: Triggers when active application count is below min_expected_apps, dispatching a Slack warning alerting SREs to profiling agent dropouts.
    • HEALTHY: Confirms ingester readiness and expected coverage, logging operational status via io.kestra.plugin.core.log.Log.
  5. log_audit_complete always records execution completion and active application counts for long-term auditability.
  6. The global errors block intercepts connection drops or socket errors to immediately notify on-call engineers when the Pyroscope container is unreachable.

What you get

  • Early warning when Pyroscope ingesters enter backpressure and drop application profile samples.
  • Proactive detection of profiling agent dropouts (e.g., following microservice redeployments or failed sidecar injections).
  • Seamless integration with Slack alerting pipelines using standard webhook endpoints.
  • Clean observability history capturing exactly when profiling coverage was compromised.

Who it's for

  • Site Reliability Engineers (SREs) and Platform Engineers maintaining observability infrastructure.
  • Performance Engineering teams relying on continuous profiling (pprof, async-profiler, eBPF) for flame graph analysis.
  • DevOps engineers managing Grafana LGTM stack deployments (Loki, Grafana, Tempo, Mimir, Pyroscope).

Why orchestrate this with Kestra

Continuous profiling is vital for root-cause latency analysis and memory leak detection, yet the profiling infrastructure itself is rarely monitored for coverage dropouts. Static ping checks cannot parse the active application array or assert whether required services are actively streaming profiles. Kestra provides declarative scheduling, robust error handling, secure credential management, and structured alerting.

Prerequisites

  • A running Grafana Pyroscope instance (version 1.0+), reachable over HTTP. Grafana publishes an official open-source Docker image (grafana/pyroscope:latest) requiring no license key or paid account.
  • A Slack incoming webhook URL configured for operational alerts.

Secrets

  • SLACK_WEBHOOK_URL: Slack incoming webhook URL used by alert_ingester_degraded, alert_coverage_deficit, and the global errors handler.

Quick start

  1. Start a local Grafana Pyroscope container:
    docker run -d --name pyroscope-test -p 4040:4040 grafana/pyroscope:latest
    
  2. Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace.
  3. Import this blueprint into Kestra and trigger an execution with default inputs (min_expected_apps: 0).
  4. Confirm that log_healthy_state logs nominal readiness.
  5. Set min_expected_apps: 5 and re-run to verify that APPLICATION_COVERAGE_DEFICIT alerts Slack when reporting services fall below threshold.
  6. Enable periodic_profiling_audit once verified.

How to extend

  • Add an array comparison step using io.kestra.plugin.core.flow.Loop to check for specific mandatory service names (e.g., checkout-service.cpu, auth-service.inuse_space).
  • Integrate with PagerDuty using io.kestra.plugin.pagerduty.notifications.PagerDutyAlert when INGESTER_DEGRADED fires in critical production environments.
  • Query Pyroscope's /metrics endpoint to monitor compactor block counts and long-term disk retention trends.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.