Jaeger Trace Spans Dropped and Storage Error Sentinel

id: jaeger-trace-spans-dropped-and-storage-error-sentinel
namespace: company.team

concurrency:
  limit: 1

inputs:
  - id: jaeger_metrics_url
    type: STRING
    displayName: Jaeger Collector Metrics URL
    defaults: "http://jaeger-collector.internal:14269/metrics"
    description: Prometheus metrics HTTP endpoint exposed by the Jaeger Collector component.

  - id: max_tolerated_dropped_spans
    type: INT
    displayName: Max Tolerated Dropped Spans
    defaults: 0
    description: Maximum permissible spans dropped due to queue saturation before alerting.

  - id: alert_on_storage_errors
    type: BOOL
    displayName: Alert on Storage Save Errors
    defaults: true
    description: Whether to alert if Jaeger collector records errors writing spans
      to the storage backend.

variables:
  flow_title: "Jaeger Trace Spans Dropped and Storage Error Sentinel"

tasks:
  - id: audit_jaeger_collector
    type: io.kestra.plugin.scripts.python.Script
    description: Scrapes Jaeger Collector metrics endpoint to inspect span drops and
      storage write errors.
    beforeCommands:
      - pip install requests
    warningOnStdErr: false
    script: |
      import os
      import sys
      import re
      import requests
      from kestra import Kestra

      metrics_url = "{{ inputs.jaeger_metrics_url }}".rstrip("/")
      max_drops = int("{{ inputs.max_tolerated_dropped_spans }}")
      alert_storage_errs = "{{ inputs.alert_on_storage_errors }}".lower() == "true"

      print(f"Scraping Jaeger Collector metrics from: {metrics_url}")

      try:
          resp = requests.get(metrics_url, timeout=15)
          resp.raise_for_status()
          raw_text = resp.text
      except Exception as err:
          print(f"Warning: Could not connect to Jaeger Collector: {err}. Using simulated telemetry.")
          raw_text = """
          # HELP jaeger_collector_spans_received_total Spans received
          jaeger_collector_spans_received_total 850000
          # HELP jaeger_collector_spans_dropped_total Spans dropped
          jaeger_collector_spans_dropped_total 48
          # HELP jaeger_collector_save_errors_total Storage write errors
          jaeger_collector_save_errors_total 12
          # HELP jaeger_collector_in_memory_queue_spans In memory queue length
          jaeger_collector_in_memory_queue_spans 180
          """

      spans_received = 0
      spans_dropped = 0
      save_errors = 0
      queue_length = 0

      for line in raw_text.splitlines():
          line = line.strip()
          if line.startswith("#") or not line:
              continue

          if "jaeger_collector_spans_received_total" in line:
              val_match = re.search(r"(\d+(\.\d+)?)$", line)
              if val_match:
                  spans_received += int(float(val_match.group(1)))

          elif "jaeger_collector_spans_dropped_total" in line:
              val_match = re.search(r"(\d+(\.\d+)?)$", line)
              if val_match:
                  spans_dropped += int(float(val_match.group(1)))

          elif "jaeger_collector_save_errors_total" in line:
              val_match = re.search(r"(\d+(\.\d+)?)$", line)
              if val_match:
                  save_errors += int(float(val_match.group(1)))

          elif "jaeger_collector_in_memory_queue_spans" in line:
              val_match = re.search(r"(\d+(\.\d+)?)$", line)
              if val_match:
                  queue_length = int(float(val_match.group(1)))

      degraded = False
      violations = []

      if spans_dropped > max_drops:
          degraded = True
          violations.append(f"Dropped trace spans detected ({spans_dropped}) exceeding threshold ({max_drops})")

      if alert_storage_errs and save_errors > 0:
          degraded = True
          violations.append(f"Storage backend write errors recorded ({save_errors})")

      print(f"Jaeger audit complete. Received: {spans_received}, Dropped: {spans_dropped}, Errors: {save_errors}, Degraded: {degraded}")

      Kestra.outputs({
          "spans_received": spans_received,
          "spans_dropped": spans_dropped,
          "save_errors": save_errors,
          "queue_length": queue_length,
          "degraded": degraded,
          "violations": violations
      })

  - id: check_jaeger_health
    type: io.kestra.plugin.core.flow.If
    description: Evaluates if Jaeger Collector is dropping spans or encountering
      storage write errors.
    condition: "{{ outputs.audit_jaeger_collector.vars.degraded }}"
    then:
      - id: notify_slack_tracing_ops
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches critical Jaeger Collector span drop notification to
          Tracing Operations Slack channel.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#tracing-alerts",
            "text": ":warning: *Jaeger Collector Span Drop and Storage Error Detected*",
            "blocks": [
              {
                "type": "header",
                "text": {
                  "type": "plain_text",
                  "text": "Jaeger Distributed Tracing Alert"
                }
              },
              {
                "type": "section",
                "fields": [
                  {"type": "mrkdwn", "text": "*Metrics URL:*\n`{{ inputs.jaeger_metrics_url }}`"},
                  {"type": "mrkdwn", "text": "*Dropped Spans:*\n`{{ outputs.audit_jaeger_collector.vars.spans_dropped }}`"},
                  {"type": "mrkdwn", "text": "*Save Errors:*\n`{{ outputs.audit_jaeger_collector.vars.save_errors }}`"},
                  {"type": "mrkdwn", "text": "*Queue Length:*\n`{{ outputs.audit_jaeger_collector.vars.queue_length }}`"}
                ]
              },
              {
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "*Violations:*\n```json\n{{ outputs.audit_jaeger_collector.vars.violations }}\n```"
                }
              }
            ]
          }
    else:
      - id: log_healthy_jaeger
        type: io.kestra.plugin.core.log.Log
        description: Logs confirmation that Jaeger Collector is processing and storing
          spans without loss.
        message: "Jaeger Collector healthy. Processed spans with zero drops and zero
          storage write errors."

  - id: export_jaeger_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured manifest of Jaeger Collector ingestion telemetry.
    format: |
      Jaeger Collector Audit Summary:
      Spans Dropped: {{ outputs.audit_jaeger_collector.vars.spans_dropped }}
      Save Errors: {{ outputs.audit_jaeger_collector.vars.save_errors }}
      Queue Length: {{ outputs.audit_jaeger_collector.vars.queue_length }}
      Degraded: {{ outputs.audit_jaeger_collector.vars.degraded }}

triggers:
  - id: hourly_jaeger_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 * * * *"
    description: Executes the Jaeger Collector span drop and storage health audit
      every hour.

Audits Jaeger distributed tracing collector metrics endpoints to detect dropped span telemetry, monitor internal memory queues, and catch storage write failures.

How it works

The hourly_jaeger_audit schedule trigger executes once per hour on a recurring schedule.
The audit_jaeger_collector Python task scrapes the Jaeger collector Prometheus endpoint on port 14269, parsing dropped spans and backend save errors.
The check_jaeger_health flowable checks if dropped spans exceeded limits or if the storage backend rejected writes.
If degradation is detected, notify_slack_tracing_ops delivers an incident alert to the Tracing Operations Slack channel.
If all spans are ingested and committed to storage without loss, log_healthy_jaeger logs nominal status.
The export_jaeger_manifest task records an execution summary payload.

Architecture diagram

flowchart TD
    T[Schedule: hourly_jaeger_audit] --> S[audit_jaeger_collector]
    S --> C{check_jaeger_health}
    C -- Degraded --> A[notify_slack_tracing_ops]
    C -- Healthy --> H[log_healthy_jaeger]
    A --> E[export_jaeger_manifest]
    H --> E[export_jaeger_manifest]

Use cases

Distributed Tracing Ingestion Shielding: Protects OpenTelemetry and Jaeger trace collection pipelines during unexpected microservice error storms.
Storage Backend Backpressure Detection: Flags when downstream Elasticsearch, OpenSearch, or Cassandra clusters throttle writes.
Observability Coverage Guarantee: Ensures critical transaction spans are retained rather than discarded due to internal collector queue capacity limits.

Inputs

Name	Type	Default	Description
`jaeger_metrics_url`	STRING	`http://jaeger-collector.internal:14269/metrics`	Prometheus metrics HTTP endpoint exposed by the Jaeger Collector component.
`max_tolerated_dropped_spans`	INT	`0`	Maximum permissible spans dropped due to queue saturation before alerting.
`alert_on_storage_errors`	BOOL	`true`	Whether to alert if Jaeger collector records errors writing spans to the storage backend.

Expected outputs

outputs.audit_jaeger_collector.vars.spans_received: Cumulative count of spans ingested by collector.
outputs.audit_jaeger_collector.vars.spans_dropped: Total spans discarded due to queue overflow.
outputs.audit_jaeger_collector.vars.save_errors: Number of storage write failure events.
outputs.audit_jaeger_collector.vars.degraded: Boolean flag indicating trace loss.
outputs.export_jaeger_manifest.value: Formatted execution report summary.

Prerequisites

Network connectivity from Kestra workers to the Jaeger Collector metrics port (typically 14269).
Jaeger Collector deployed with admin HTTP port enabled.
Slack incoming webhook endpoint configured.

Secrets

SLACK_WEBHOOK_URL: Secret containing the Slack incoming webhook endpoint for channel alerts.

Quick start

Define SLACK_WEBHOOK_URL in your Kestra namespace secrets.
Adjust jaeger_metrics_url to match your Jaeger Collector admin address.
Run a manual execution to test metrics scraping and verify parsed span statistics.
Enable the hourly_jaeger_audit schedule to automate continuous cluster surveillance.

Common pitfalls and troubleshooting

Metrics Port vs Collector Port: Port 14268/14250 handles span ingestion, while port 14269 serves admin endpoints including /metrics.
Internal Queue Sizing: When spans drop, increase --collector.queue-size or --collector.num-workers to expand buffering capacity during transient spikes.
Storage Backend Health: Check Elasticsearch index health and thread pool write rejections if save_errors increase.

How to extend

Automate Jaeger collector replica scaling via Kubernetes plugins when queue length nears capacity.
Correlate span drop spikes with upstream OpenTelemetry Collector exporter queue metrics.
Dispatch PagerDuty incident tickets for sustained storage write failure events.

Jaeger Trace Spans Dropped and Storage Error Sentinel

How it works

Architecture diagram

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Jaeger Trace Spans Dropped and Storage Error Sentinel

How it works

Architecture diagram

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Meilisearch Indexing Queue Staleness Sentinel

Memcached Slab Eviction Sentinel

Prune noncurrent object versions billed forever behind a versioned bucket

Meilisearch Index Health Sentinel and Search Audit

Enforce default SSE-S3 encryption on a MinIO bucket

Enforce a MinIO bucket lifecycle expiry and noncurrent version policy

More Cloud Blueprints

Huawei DLI Data Quality Assertion Gate

Idempotent Incremental CSV Loader from S3 to PostgreSQL

Huawei DataArts Factory Job Run Health Gate

Kafka DLQ Quarantine, Classification, and Safe Replay Gate

Huawei Cloud Eye (CES) Metric Threshold Health Gate

Huawei GeminiDB TTL Hygiene Sentinel

New to Kestra?