Schedule icon
Request icon
Script icon
Docker icon
Switch icon
SlackIncomingWebhook icon
Log icon
OutputValues icon

SigNoz OpenTelemetry Collector & Ingestion Sentinel

Audits SigNoz OpenTelemetry collector buffer saturation, detects dropped spans, and triggers automated Slack alerts with capacity remediation runbooks.

Categories
Infrastructure

SigNoz is an open-source, full-stack APM and observability platform that ingests distributed traces, system metrics, and application logs using the OpenTelemetry Collector and ClickHouse storage engine. During high-traffic events, bursty telemetry loads, or downstream database write contention, collector exporter queues can saturate. When buffer capacities are exceeded, the collector begins dropping spans and rejecting incoming trace batches, leading to permanent blind spots in production monitoring.

This blueprint provides an automated observability sentinel that periodically scrapes the SigNoz OpenTelemetry collector internal Prometheus metrics endpoint, calculates exporter queue saturation percentages against capacity limits, inspects dropped and refused span counters, routes status through a stateful classification gate, dispatches actionable Slack incident cards, and exports structured telemetry for SRE capacity planning.

How it works

  1. periodic_collector_audit (io.kestra.plugin.core.trigger.Schedule) initiates automated surveillance every ten minutes. It is shipped disabled by default for initial connection verification.
  2. probe_query_service_health (io.kestra.plugin.core.http.Request) checks the SigNoz Query Service API /api/v1/health endpoint to verify the backend query layer is responsive.
  3. probe_collector_metrics (io.kestra.plugin.core.http.Request) scrapes the OpenTelemetry collector internal Prometheus telemetry endpoint GET /metrics on port 8888.
  4. evaluate_ingestion_buffer (io.kestra.plugin.scripts.python.Script running inside Docker) parses Prometheus exposition metrics, sums otelcol_exporter_queue_size and otelcol_exporter_queue_capacity, tracks otelcol_receiver_refused_spans and otelcol_exporter_enqueue_failed_spans, and classifies system state.
  5. classify_ingestion_state (io.kestra.plugin.core.flow.Switch) evaluates the parsed saturation figures:
    • CRITICAL_INGESTION_SATURATION: Triggered when queue saturation meets or exceeds 90% or any dropped/refused spans are detected. Dispatches an urgent incident briefing card to #observability-alerts via io.kestra.plugin.slack.notifications.SlackIncomingWebhook containing exact queue metrics and ClickHouse remediation guidance.
    • BUFFER_QUEUE_WARNING: Triggered when queue depth exceeds queue_threshold_pct (default 75%). Sends an advisory alert for proactive capacity scaling.
    • HEALTHY: Records nominal queue metrics via io.kestra.plugin.core.log.Log when buffers operate well within safe margins.
  6. finalize_ingestion_telemetry (io.kestra.plugin.core.output.OutputValues) publishes clean audit telemetry for operational dashboards and capacity audits.
  7. The global errors handler captures connection drops or crashed collector daemons to alert SRE teams out-of-band.

What you get

  • Early warning of collector exporter buffer exhaustion before production traces and logs are dropped.
  • Continuous validation of SigNoz Query Service uptime and OpenTelemetry collector responsiveness.
  • Contextual Slack alerts containing exact queue utilization percentages, buffer depths, and SRE runbook steps.
  • Structured telemetry records preserved in Kestra execution history for capacity planning.

Who it's for

  • Site Reliability Engineers (SREs) and Platform Engineers operating SigNoz clusters in Kubernetes and Docker environments.
  • Observability teams responsible for telemetry pipeline availability and telemetry SLA enforcement.
  • DevOps engineers managing OpenTelemetry ingestion gateways across distributed microservices.

Why orchestrate this with Kestra

Observability pipelines require reliable scheduled polling, containerized metric parsing, secure secret masking, conditional escalation, and resilient error recovery. Standalone bash or python cron scripts lack persistent execution history, centralized audit logs, and unified alert formatting. Kestra provides declarative YAML workflows, built-in secret masking, robust error routing, and real-time execution observability.

Prerequisites

  • A running instance of SigNoz, reachable over HTTP. SigNoz provides official open-source Docker images (signoz/signoz:latest and signoz/signoz-otel-collector:latest) with no commercial license required.
  • OpenTelemetry collector internal metrics port (default 8888) exposed to the network.
  • An incoming Slack webhook configured for operational alerts.
  • Docker enabled on Kestra workers to execute the Python metric parsing container.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used to dispatch operational and saturation alerts.

Quick start

  1. Start a local SigNoz standalone installation or spin up the OpenTelemetry collector container:
    docker run -d --name signoz-collector -p 8888:8888 signoz/signoz-otel-collector:latest
    
  2. Configure SLACK_WEBHOOK_URL in your Kestra namespace secrets.
  3. Import this blueprint into Kestra under namespace company.observability.
  4. Click Execute with default parameters (signoz_collector_url: http://localhost:8888) to confirm metric scraping and observe healthy telemetry outputs.
  5. Enable periodic_collector_audit to activate continuous ten-minute ingestion surveillance.

How to extend

  • Add automated horizontal scaling: invoke Kubernetes API tasks (io.kestra.plugin.kubernetes.pods.Patch) to scale OpenTelemetry collector replicas automatically when buffer queue warnings occur.
  • Integrate with PagerDuty via io.kestra.plugin.pagerduty.notifications.PagerDutyAlert for Sev-1 paging when dropped spans exceed 100 within a polling window.
  • Export collector buffer saturation metrics to long-term storage or ClickHouse for historical capacity trend forecasting.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.