Script icon
Docker icon
If icon
SlackIncomingWebhook icon
Log icon
Return icon
Schedule icon

Uptrace Tracing Ingestion Queue and Dropped Spans Sentinel

Audits Uptrace APM Prometheus metrics to detect ingestion queue backpressure, dropped trace spans, and ClickHouse write latencies.

Categories
CloudInfrastructure

Automates continuous health and performance surveillance for Uptrace open-source APM distributed tracing backends.

How it works

  1. A schedule trigger executes the Uptrace APM audit workflow every 15 minutes.
  2. The Python task queries the Uptrace Prometheus metrics endpoint on port 14318.
  3. The script extracts processed trace spans, dropped span records, memory ingestion queue depths, and ClickHouse write latencies.
  4. An evaluation gate determines whether dropped span volumes or database latencies breach operational tolerance bounds.
  5. When thresholds are breached, a high-priority Slack alert is dispatched to observability engineering and APM SREs.
  6. When spans ingest cleanly and ClickHouse commits without lag, nominal telemetry is logged.
  7. A structured JSON audit manifest is emitted for APM telemetry governance and tracing SLA compliance records.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 15m] --> ScanMetrics[Python Script: Scrape Uptrace Metrics]
    ScanMetrics --> EvalBreach{Dropped Spans > Threshold OR Latency > 300ms?}
    EvalBreach -- Yes --> SlackAlert[Slack: High-Urgency Observability Alert]
    EvalBreach -- No --> LogNominal[Log: Nominal Uptrace State]
    SlackAlert --> EmitManifest[Debug Return: Uptrace Audit Manifest]
    LogNominal --> EmitManifest

Architecture components

  • Trigger: Automated schedule polling at 15-minute intervals.
  • Script Task: Scrapes Uptrace Prometheus telemetry endpoints using Docker container isolation.
  • Decision Gate: Evaluates whether dropped span counts or storage write latencies exceed safety limits.
  • Notification Task: Dispatches structured Slack alerts with environment identity and error counts.
  • Audit Manifest: Emits JSON execution payloads containing APM ingestion health and timestamps.

Key capabilities

  • Continuous surveillance of Uptrace OpenTelemetry ingestion buffers and span batch drop counts.
  • Rapid detection of ClickHouse storage insert latencies and disk write throttling stalls.
  • Automated tracking of in-memory ingestion queue lengths during heavy microservice load bursts.
  • Zero external dependency execution via isolated containerized Python runners.

What you get

  • Preemptive alerting on distributed tracing bottlenecks before application performance visibility is blinded.
  • Automated tracking of ClickHouse storage write latency and trace buffer overflows.
  • Real-time audit manifests capturing APM cluster health for SOC 2 observability compliance.
  • Configurable tolerance thresholds adaptable to high-throughput containerized microservices.

Who it's for

  • Observability Platform SREs operating self-hosted Uptrace and ClickHouse APM backends.
  • Site Reliability Engineers relying on distributed tracing context to isolate production outages.
  • Performance Architects monitoring OpenTelemetry collector throughput and batch processor efficiency.
  • DevOps Teams diagnosing trace sample drops and application performance monitoring gaps.

Why orchestrate this with Kestra

  • Deterministic Scheduling: Guarantees regular APM pipeline health audits without fragile local crontab scripts.
  • Containerized Runners: Runs Python tasks inside isolated Docker containers without host library dependencies.
  • Condition Branching: Flowable If tasks route alerts and nominal logging cleanly without spaghetti code.
  • Enterprise Secret Safety: Securely injects Slack webhook credentials without plain-text exposure.

Use cases

  • Tracing Buffer Overflow Alerting: Alerts operators when bursts of service requests fill Uptrace memory queues.
  • ClickHouse Write Throttling Detection: Identifies when slow disk merges in ClickHouse cause span ingestion to stall.
  • OpenTelemetry Drop Rate Monitoring: Validates that critical microservice spans are ingested without sampling degradation.
  • Continuous APM SLA Auditing: Emits periodic manifests proving high span ingestion availability and low latency.

Inputs

Name Type Default Description
uptrace_metrics_url STRING http://uptrace.monitoring.svc.cluster.local:14318 Base HTTP URL exposing Uptrace metrics.
target_environment STRING production-uptrace-apm Target Uptrace APM environment identifier.
max_dropped_spans_threshold INT 25 Max allowable dropped span records in an audit window before alerting.
max_clickhouse_latency_ms INT 300 Maximum acceptable ClickHouse write and query latency.
alert_on_any_queue_overflow BOOL false Whether any nonzero queue overflow triggers an immediate alert.

Expected outputs

  • {{ outputs.scan_uptrace_telemetry.vars.spans_processed_total }}: Total trace spans processed by Uptrace.
  • {{ outputs.scan_uptrace_telemetry.vars.spans_dropped_total }}: Count of dropped spans due to buffer backpressure.
  • {{ outputs.scan_uptrace_telemetry.vars.queue_length_gauge }}: Current in-memory ingestion queue length.
  • {{ outputs.scan_uptrace_telemetry.vars.storage_latency_ms }}: ClickHouse trace storage commit latency in milliseconds.
  • {{ outputs.scan_uptrace_telemetry.vars.is_alerting }}: Boolean flag indicating ingestion threshold violation.
  • {{ outputs.emit_audit_manifest.value }}: Structured JSON audit manifest emitted for compliance archives.

Prerequisites

  • Uptrace deployed with Prometheus metrics enabled on port 14318 (metrics.addr).
  • Metrics port reachable from the Kestra worker execution environment.
  • Docker daemon running on the Kestra host to execute containerized Python tasks.

Secrets

  • SLACK_WEBHOOK_URL: Incoming webhook URL for the target Observability SRE Slack alerting channel.

Quick start

  1. Ensure Uptrace service exposes port 14318 for Prometheus metrics scraping.
  2. Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace settings.
  3. Specify your environment identifier in target_environment.
  4. Execute a manual test run via the Kestra UI to verify metric ingestion.
  5. Enable the periodic_uptrace_audit trigger to establish automated continuous surveillance.

Common pitfalls and troubleshooting

  • Port Binding Configuration: By default, Uptrace binds metrics to 0.0.0.0:14318; verify NetworkPolicies permit ingress.
  • ClickHouse Parts Merges: High ClickHouse write latency often correlates with unmerged data parts in trace tables.
  • Queue Sizing in YAML: Low queue_size settings in uptrace.yml cause premature span dropouts under traffic spikes.
  • Memory Limits: Heavy unbuffered span volume can trigger OOM kill events on Uptrace collector containers.

How to extend

  • Automated ClickHouse Optimization: Trigger an OPTIMIZE TABLE task in ClickHouse when unmerged parts slow ingestion.
  • Dynamic Collector Tuning: Adjust OpenTelemetry batch timeout and buffer queue settings via config discovery.
  • PagerDuty Escalation: Escalate to on-call observability engineers if span drops exceed 500 in 15 minutes.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.