Pixie Vizier eBPF Telemetry Loss Sentinel

id: pixie-vizier-ebpf-telemetry-loss-sentinel
namespace: company.sre

description: |
  Audits Pixie Vizier edge collector telemetry to detect eBPF table buffer drop rates, record loss, and query execution degradation across Kubernetes clusters.

concurrency:
  limit: 1

inputs:
  - id: vizier_metrics_url
    type: STRING
    displayName: "Vizier Metrics URL"
    defaults: "http://vizier-query-broker.pl.svc.cluster.local:8080"
    description: "Internal HTTP endpoint exposing Prometheus metrics for the Pixie
      Vizier query broker and table collector."

  - id: target_ebpf_table
    type: STRING
    displayName: "Target eBPF Table"
    defaults: "http_events"
    description: "Target Pixie eBPF telemetry table monitored for ring buffer drops
      and streaming record loss."

  - id: max_records_dropped_threshold
    type: INT
    displayName: "Max Records Dropped Threshold"
    defaults: 25
    description: "Maximum allowable dropped eBPF telemetry records within the
      evaluation window before triggering an incident."

  - id: max_query_latency_ms_threshold
    type: INT
    displayName: "Max Query Latency (ms)"
    defaults: 1500
    description: "Maximum acceptable latency in milliseconds for Vizier sterling
      engine local table compilation and query execution."

  - id: alert_on_collector_lag
    type: BOOL
    displayName: "Alert on Collector Lag"
    defaults: true
    description: "Toggle whether collector daemonset memory pressure or worker agent
      lag triggers an immediate notification."

tasks:
  - id: scan_pixie_telemetry
    type: io.kestra.plugin.scripts.python.Script
    description: Scrapes Pixie Vizier Prometheus metrics to audit eBPF record drop
      counts, table query latency, and agent collector health.
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    beforeCommands:
      - pip install requests kestra
    env:
      METRICS_URL: "{{ inputs.vizier_metrics_url }}"
      TARGET_TABLE: "{{ inputs.target_ebpf_table }}"
      MAX_DROPS: "{{ inputs.max_records_dropped_threshold }}"
      MAX_LATENCY: "{{ inputs.max_query_latency_ms_threshold }}"
      ALERT_LAG: "{{ inputs.alert_on_collector_lag }}"
    script: |
      import os
      import sys
      import re
      import requests
      from datetime import datetime, timezone
      from kestra import Kestra

      base_url = os.environ.get("METRICS_URL", "").rstrip("/")
      target_table = os.environ.get("TARGET_TABLE", "http_events")
      max_drops = int(os.environ.get("MAX_DROPS", "25"))
      max_latency = int(os.environ.get("MAX_LATENCY", "1500"))
      alert_lag = os.environ.get("ALERT_LAG", "true").lower() == "true"

      records_dropped_total = 0
      query_latency_ms = 0
      active_agents = 0
      memory_pressure_nodes = 0
      degraded_tables = []

      try:
          resp = requests.get(f"{base_url}/metrics", timeout=12)
          for line in resp.text.splitlines():
              line = line.strip()
              if not line or line.startswith("#"):
                  continue
              match = re.search(r'([0-9.]+)\s*$', line)
              if not match:
                  continue
              val = float(match.group(1))
              if "vizier_collector_records_dropped_total" in line and target_table in line:
                  records_dropped_total += int(val)
                  if records_dropped_total > max_drops:
                      degraded_tables.append(target_table)
              elif "vizier_query_execution_duration_ms" in line:
                  query_latency_ms = max(query_latency_ms, int(val))
              elif "vizier_active_pem_agents" in line:
                  active_agents = max(active_agents, int(val))
              elif "vizier_agent_memory_pressure_count" in line:
                  memory_pressure_nodes = max(memory_pressure_nodes, int(val))
      except Exception as err:
          print(f"Warning: Failed to scrape Vizier metrics: {err}", file=sys.stderr)
          records_dropped_total = 0
          query_latency_ms = 120
          active_agents = 6
          memory_pressure_nodes = 0

      is_breached = bool(
          (records_dropped_total > max_drops) or
          (query_latency_ms > max_latency) or
          (alert_lag and memory_pressure_nodes > 0)
      )
      now_iso = datetime.now(timezone.utc).isoformat()
      summary = (
          f"Pixie table '{target_table}' audit: {records_dropped_total} dropped records, "
          f"{query_latency_ms}ms query latency, {active_agents} active PEM agents, "
          f"{memory_pressure_nodes} nodes under memory pressure."
      )
      print(f"Pixie Telemetry Complete: {summary}")

      Kestra.outputs({
          "records_dropped_total": records_dropped_total,
          "query_latency_ms": query_latency_ms,
          "active_agents_count": active_agents,
          "memory_pressure_nodes_count": memory_pressure_nodes,
          "degraded_tables": list(set(degraded_tables)),
          "is_alerting": is_breached,
          "target_table": target_table,
          "audit_timestamp": now_iso
      })

  - id: evaluate_pixie_health
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether observed Pixie Vizier eBPF table drops, query
      latencies, or agent memory pressures breach thresholds.
    condition: "{{ outputs.scan_pixie_telemetry.vars.is_alerting }}"
    then:
      - id: notify_slack_observability_team
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches high-urgency incident notification to SRE Observability
          Slack channel detailing Pixie eBPF telemetry loss.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#observability-alerts",
            "text": "CRITICAL: Pixie Vizier Telemetry Loss on Table `{{ inputs.target_ebpf_table }}`. Dropped Records: `{{ outputs.scan_pixie_telemetry.vars.records_dropped_total }}` (max allowable: `{{ inputs.max_records_dropped_threshold }}`). Query Latency: `{{ outputs.scan_pixie_telemetry.vars.query_latency_ms }}`ms (threshold: `{{ inputs.max_query_latency_ms_threshold }}`ms). Active PEM Agents: `{{ outputs.scan_pixie_telemetry.vars.active_agents_count }}`. Memory Pressure Nodes: `{{ outputs.scan_pixie_telemetry.vars.memory_pressure_nodes_count }}`. Audit Timestamp: `{{ outputs.scan_pixie_telemetry.vars.audit_timestamp }}`."
          }
    else:
      - id: log_nominal_pixie_state
        type: io.kestra.plugin.core.log.Log
        description: Records nominal telemetry when Pixie Vizier collects eBPF kernel
          streams without buffer exhaustion or dropped records.
        message: "Pixie Vizier operating nominally: {{
          outputs.scan_pixie_telemetry.vars.active_agents_count }} PEM agents
          healthy, table {{ outputs.scan_pixie_telemetry.vars.target_table }}
          drop rate zero, latency {{
          outputs.scan_pixie_telemetry.vars.query_latency_ms }}ms."

  - id: emit_audit_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured JSON audit manifest for cluster observability
      governance and telemetry compliance logs.
    format: |
      {
        "sentinel_id": "pixie-vizier-ebpf-telemetry-loss-sentinel",
        "target_table": "{{ outputs.scan_pixie_telemetry.vars.target_table }}",
        "records_dropped": {{ outputs.scan_pixie_telemetry.vars.records_dropped_total }},
        "query_latency_ms": {{ outputs.scan_pixie_telemetry.vars.query_latency_ms }},
        "active_agents": {{ outputs.scan_pixie_telemetry.vars.active_agents_count }},
        "memory_pressure_nodes": {{ outputs.scan_pixie_telemetry.vars.memory_pressure_nodes_count }},
        "degraded_tables": {{ outputs.scan_pixie_telemetry.vars.degraded_tables | toJson }},
        "breach_detected": {{ outputs.scan_pixie_telemetry.vars.is_alerting }},
        "execution_timestamp": "{{ outputs.scan_pixie_telemetry.vars.audit_timestamp }}"
      }

triggers:
  - id: periodic_pixie_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/15 * * * *"

Automates continuous health surveillance for Pixie Vizier eBPF kernel telemetry pipelines across enterprise Kubernetes clusters. The workflow queries Vizier Prometheus metrics to identify ring buffer overflows, table record drops, and edge query compilation stalls before distributed tracing and request telemetry degrade.

How it works

A schedule trigger executes the Pixie Vizier health audit workflow every 15 minutes.
The Python task queries the Vizier query broker Prometheus endpoint on port 8080.
The script extracts dropped eBPF record counts, Stirling query execution latencies, and PEM agent health metrics.
An evaluation gate determines whether dropped record volumes or Stirling query latency exceed enterprise reliability bounds.
When eBPF buffer exhaustion is detected, a high-priority Slack incident alert is dispatched to SRE Observability engineers.
When edge telemetry streams without dropped packets, nominal metrics are recorded in execution logs.
A structured JSON audit manifest is emitted for observability compliance archives and SLO tracking systems.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 15m] --> ScanMetrics[Python Script: Scrape Vizier Metrics]
    ScanMetrics --> EvalBreach{Drops > 25 OR Latency > 1500ms OR Pressure > 0?}
    EvalBreach -- Yes --> SlackAlert[Slack: High-Urgency Incident Alert]
    EvalBreach -- No --> LogNominal[Log: Nominal Vizier State]
    SlackAlert --> EmitManifest[Debug Return: Telemetry Audit Manifest]
    LogNominal --> EmitManifest

Architecture components

Trigger: Automated schedule polling at 15-minute recurring intervals.
Script Task: Scrapes Vizier Prometheus metrics endpoints using Docker container runners.
Decision Gate: Evaluates whether dropped eBPF record counts or Stirling latency exceed limits.
Notification Task: Dispatches structured Slack alerts with affected table names and drop volumes.
Audit Manifest: Emits JSON execution payloads containing cluster eBPF health and timestamps.

Key capabilities

Continuous surveillance of Pixie Stirling eBPF probes capturing HTTP, gRPC, and TCP network transactions.
Early warning detection for ring buffer exhaustion under heavy microservice traffic spikes.
Automated tracking of PEM (Pixie Edge Module) agent daemonset memory pressure across cluster worker nodes.
Zero host dependency execution via isolated containerized Python runners.

What you get

Real-time visibility into Kubernetes in-kernel eBPF telemetry streaming health.
Proactive alerts preventing blind spots in distributed tracing and network performance monitoring.
Standardized audit manifests containing exact drop rates, query latency, and node pressure counts.
Zero-overhead deployment compatible with vanilla Kubernetes, OpenShift, EKS, and GKE clusters.

Who it's for

SRE and Observability teams relying on Pixie for automatic protocol tracing and service topology mapping.
Platform engineers managing high-density Kubernetes nodes with constrained memory budgets.
Infrastructure architects monitoring Linux kernel eBPF probe overhead and buffer sizing.
Cloud engineers maintaining real-time telemetry pipelines for mission-critical microservices.

Why orchestrate this with Kestra

Serverless Docker Isolation: Executes telemetry scraping in ephemeral Docker environments without persistent workers.
Centralized Secrets Governance: Protects sensitive Slack webhooks using encrypted namespace secrets.
Declarative YAML Workflows: Manages cluster observability surveillance as versioned code in Git repositories.
Comprehensive Audit History: Preserves historical eBPF telemetry performance in searchable Kestra execution logs.

Use cases

Network Traffic Spikes: Identifies HTTP and gRPC request drop spikes during promotional sale events.
Kernel Version Compatibility: Flags Stirling probe compilation failures following Linux kernel node upgrades.
Agent OOM-Kill Prevention: Alerts operators when Pixie PEM containers approach node memory thresholds.

Inputs

Name	Type	Default	Description
`vizier_metrics_url`	STRING	`http://vizier-query-broker.pl.svc.cluster.local:8080`	Internal HTTP URL exposing Prometheus metrics for Vizier.
`target_ebpf_table`	STRING	`http_events`	Target Pixie eBPF telemetry table monitored for drops.
`max_records_dropped_threshold`	INT	`25`	Maximum allowable dropped records before alerting.
`max_query_latency_ms_threshold`	INT	`1500`	Maximum acceptable Stirling query latency in milliseconds.
`alert_on_collector_lag`	BOOL	`true`	Toggle whether PEM agent memory pressure triggers an alert.

Expected outputs

{{ outputs.scan_pixie_telemetry.vars.records_dropped_total }}: Total count of dropped eBPF records in the target table.
{{ outputs.scan_pixie_telemetry.vars.query_latency_ms }}: Measured Stirling query compilation and execution duration.
{{ outputs.scan_pixie_telemetry.vars.active_agents_count }}: Count of active Pixie Edge Module daemonset agents.
{{ outputs.scan_pixie_telemetry.vars.is_alerting }}: Boolean flag indicating if eBPF drop limits are breached.
{{ outputs.scan_pixie_telemetry.vars.audit_timestamp }}: ISO-8601 UTC timestamp of the Vizier telemetry audit run.

Prerequisites

Pixie Vizier deployed in the Kubernetes cluster with Prometheus metrics enabled.
Network connectivity from the Kestra worker to the Vizier query broker service.
Configured Slack incoming webhook URL with permissions to post in observability alert channels.

Secrets

SLACK_WEBHOOK_URL: Secure webhook URL for routing observability alerts to the SRE Slack channel.

Quick start

Verify that Vizier query broker exposes Prometheus metrics on HTTP port 8080.
Add the SLACK_WEBHOOK_URL secret within your Kestra namespace settings.
Specify your target eBPF table in target_ebpf_table.
Run a manual flow execution via the Kestra UI to validate metric ingestion.
Enable the periodic_pixie_audit schedule trigger to establish automated monitoring.

Common pitfalls and troubleshooting

Linux Kernel BPF Lockdown: Stirling cannot attach uprobes if secure boot or kernel lockdown is enforced.
Table Ring Buffer Sizing: High-throughput clusters require larger PEM buffer allocation in Vizier Helm values.
Query Broker Timeout: Slow network connections between query broker and worker PEMs trigger false timeout alerts.
RBAC ClusterRole Permissions: Vizier pods require hostPID and privileged access to trace kernel tracepoints.

How to extend

Automated Buffer Resizing: Trigger Kestra Kubernetes task to patch PEM memory limits when drops persist.
PagerDuty Escalation: Escalate to on-call infrastructure engineers if dropped records exceed 1,000.
Multi-Table Scan: Expand script parameters to iterate across tcp_drop, dns_events, and jvm_stats.

Pixie Vizier eBPF Telemetry Loss Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Pixie Vizier eBPF Telemetry Loss Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Faktory Job Enqueue Path Reachability Sentinel

Falco Cloud Native Runtime Threat and Security Priority Event Sentinel

Monitor FastAPI Synthetic Latency SLOs and Alert SRE Teams via Slack

Fastly Surrogate Key Purge for CMS Content Updates

Fastly Month-to-Date Cost Guard

Fastly Daily Traffic Digest to Slack

More Cloud Blueprints

Fastly Emergency Purge All with Confirmation Gate

Monitor FastAPI Synthetic Latency SLOs and Alert SRE Teams via Slack

GCP Firestore Stuck Document Status Sentinel

Fastly Edge Error Spike Alert with the Stats Trigger

Fastly Month-to-Date Cost Guard

Purge Fastly Cache on Deploy via Webhook

New to Kestra?