Request icon
Reader icon
OutputValues icon
Switch icon
Log icon
SlackIncomingWebhook icon
If icon
Produce icon
Schedule icon
Webhook icon

Apache Pulsar Topic Compaction Backlog and Retention Guard

Audit Apache Pulsar key-value changelog topics for uncompacted BookKeeper ledger bloat, trigger on-demand compaction, and verify state convergence with Kestra.

Categories
Data

Apache Pulsar provides native Topic Compaction for key-value changelog streams (such as event sourcing logs, table state caches, and CDC pipelines). In compacted topics, Pulsar retains only the most recent message for each key, writing compacted records into a dedicated BookKeeper ledger while preserving the original immutable backlog for standard streaming consumers.

When topic compaction fails or lags behind high-throughput write volumes, uncompacted ledgers accumulate rapidly across BookKeeper bookies. Downstream subscribers reading from compacted subscriptions (readCompacted: true) suffer severe read latency amplification having to scan millions of superseded duplicate records, while disk utilization inflates. This blueprint audits compaction execution state and ledger bloat across Pulsar topics, classifies compaction health, safely triggers on-demand compaction via the Pulsar Admin REST API, and publishes an audit canary to verify state convergence.

How it works

  1. Scheduled or Webhook Trigger: The workflow executes on a daily maintenance schedule (0 2 * * *, shipped disabled for safe onboarding) or immediately via an authenticated webhook trigger called by Prometheus Alertmanager.
  2. Compaction Status Introspection: The probe_compaction_status task (io.kestra.plugin.core.http.Request) invokes the Pulsar Admin REST API (GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/compaction) to determine whether compaction is SUCCESS, RUNNING, or ERROR.
  3. Ledger Storage Telemetry: The probe_topic_stats task (io.kestra.plugin.core.http.Request) inspects topic metrics (GET /stats) to retrieve raw storageSize.
  4. Compacted Ledger Size: The probe_compacted_ledger task (io.kestra.plugin.core.http.Request) queries GET /internalStats for compactedLedger.size -- the real byte size of the compacted ledger. (The /stats endpoint's compaction object only exposes event counts and timestamps, never a byte size.)
  5. Broker Reachability Sampling: The probe_topic_reader task (io.kestra.plugin.pulsar.Reader) samples recent messages over the native binary protocol (pulsar://<broker>:6650) using a non-durable reader to verify broker connectivity without advancing consumer group cursors.
  6. Compaction Health Classification: The evaluate_compaction_health task (io.kestra.plugin.core.output.OutputValues) computes the uncompacted ledger volume (storageSize minus compactedLedger.size) and classifies topic health into HEALTHY, COMPACTION_IN_PROGRESS, COMPACTION_FAILED, or COMPACTION_OVERDUE.
  7. Flow Control & Remediation: The route_by_compaction_state task (io.kestra.plugin.core.flow.Switch) branches on the evaluated status:
    • HEALTHY: Emits an audit log confirming ledger storage is within acceptable thresholds.
    • COMPACTION_IN_PROGRESS: Logs an advisory note that background compaction is currently active, skipping redundant commands.
    • COMPACTION_FAILED: Alerts the streaming operations team on Slack with diagnostic fields.
    • COMPACTION_OVERDUE: Alerts Slack with uncompacted backlog volume. If auto_remediate is true and dry_run is false, it triggers on-demand topic compaction via PUT /compaction, publishes a checkpoint canary record via io.kestra.plugin.pulsar.Produce, and logs the trigger outcome.
  8. Top-Level Error Safeguards: Communication failures with the Pulsar REST endpoint or binary broker protocol trigger the top-level errors handler, notifying the on-call team.

What you get

  • Automated Changelog Compaction Governance: Prevents uncompacted BookKeeper ledger runaway on high-volume key-value topics without manual pulsar-admin CLI sessions.
  • Compacted Consumer Read Latency Protection: Ensures compacted subscriptions scan only deduplicated state ledgers, eliminating read throughput bottlenecks.
  • Dual-Plane Validation: Combines Pulsar Admin REST control-plane telemetry with live broker binary protocol sampling via io.kestra.plugin.pulsar.Reader.
  • Safe Dry-Run Safety Gates: Defaults to auto_remediate: false and dry_run: true so operators can inspect diagnostic reports before enabling automatic compaction triggers.
  • Audit Checkpoint Tracking: Publishes an auditable checkpoint record via io.kestra.plugin.pulsar.Produce to establish deterministic boundaries for compaction cycles.

Who it's for

  • Streaming Data Platform Engineers maintaining mission-critical Apache Pulsar clusters running stateful applications and CDC ingest.
  • Site Reliability Engineers (SREs) overseeing BookKeeper storage quotas, broker memory limits, and streaming consumer SLAs.
  • Event-Driven Microservice Developers relying on compacted changelog topics for local cache rehydration.

Why orchestrate this with Kestra

Apache Pulsar provides topic compaction mechanics, but does not provide an external health orchestrator to monitor compaction failures, classify ledger growth trends, or coordinate multi-step remediation with notification pipelines. Ad-hoc bash cron scripts lack centralized execution tracking, credential masking, and failure alerting.

Kestra bridges this gap by declaring the inspection, classification, remediation, and alerting sequence in version-controlled YAML. Kestra manages JWT tokens securely via the secret store, isolates network calls with declarative retries, and maintains a complete execution history for compliance and operational reviews.

Prerequisites

  • A running Apache Pulsar cluster (v2.10.x or v3.x) with Admin REST API (port 8080) and Broker binary protocol (port 6650) accessible from Kestra worker nodes. Locally testable via the official apachepulsar/pulsar:latest Docker image.
  • An active persistent topic configured with keyed messages.
  • An administrative JWT token or superuser credentials for the Pulsar cluster.
  • A Slack Incoming Webhook URL for incident alerting.

Secrets

  • PULSAR_AUTH_TOKEN: JWT Bearer token authorized to perform administrative operations and read/produce messages on the target Pulsar tenant.
  • PULSAR_GUARD_WEBHOOK_KEY: Secret authentication key securing the external Kestra webhook trigger.
  • SLACK_WEBHOOK_URL: HTTPS endpoint for delivering structured incident alerts and remediation summaries to Slack.

Quick start

  1. Configure Secrets: In Kestra, navigate to Administration -> Secrets and create PULSAR_AUTH_TOKEN, PULSAR_GUARD_WEBHOOK_KEY, and SLACK_WEBHOOK_URL.
  2. Start Local Pulsar Container: If validating locally, launch Apache Pulsar in standalone mode:
    docker run -d --name pulsar-standalone -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:3.2.0 bin/pulsar standalone
    
  3. Run Flow in Dry-Run Mode: Click Execute in Kestra with dry_run: true and auto_remediate: false. Verify that probe_compaction_status and probe_topic_stats fetch live metrics without errors.
  4. Enable Scheduled Monitoring: Toggle disabled: false on daily_maintenance_schedule once endpoints and thresholds are confirmed.

How to extend

  • Prometheus Metric Export: Add an io.kestra.plugin.core.http.Request task to export uncompacted backlog bytes and compaction duration to Prometheus Pushgateway.
  • Compaction Threshold Tuning: Automatically update namespace-level compactionThreshold via POST /admin/v2/namespaces/{tenant}/{namespace}/compactionThreshold based on daily traffic volatility.
  • Dead-Letter Integration: Connect this workflow with flows/pulsar-dead-letter-topic-sentinel.yaml to create an end-to-end Pulsar queue hygiene suite.
  • PagerDuty Escalation: Route COMPACTION_FAILED alerts to PagerDuty via io.kestra.plugin.pagerduty.notifications.PagerDutyAlert for 24/7 on-call paging.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.