Pulsar Topic Activity Sentinel
Non-destructively sample a Pulsar topic on a schedule with a non-durable reader and page Slack when it goes silent, floods, or the broker is unreachable.
Audit Apache Pulsar key-value changelog topics for uncompacted BookKeeper ledger bloat, trigger on-demand compaction, and verify state convergence with Kestra.
Apache Pulsar provides native Topic Compaction for key-value changelog streams (such as event sourcing logs, table state caches, and CDC pipelines). In compacted topics, Pulsar retains only the most recent message for each key, writing compacted records into a dedicated BookKeeper ledger while preserving the original immutable backlog for standard streaming consumers.
When topic compaction fails or lags behind high-throughput write volumes, uncompacted ledgers accumulate rapidly across BookKeeper bookies. Downstream subscribers reading from compacted subscriptions (readCompacted: true) suffer severe read latency amplification having to scan millions of superseded duplicate records, while disk utilization inflates. This blueprint audits compaction execution state and ledger bloat across Pulsar topics, classifies compaction health, safely triggers on-demand compaction via the Pulsar Admin REST API, and publishes an audit canary to verify state convergence.
0 2 * * *, shipped disabled for safe onboarding) or immediately via an authenticated webhook trigger called by Prometheus Alertmanager.probe_compaction_status task (io.kestra.plugin.core.http.Request) invokes the Pulsar Admin REST API (GET /admin/v2/persistent/{tenant}/{namespace}/{topic}/compaction) to determine whether compaction is SUCCESS, RUNNING, or ERROR.probe_topic_stats task (io.kestra.plugin.core.http.Request) inspects topic metrics (GET /stats) to retrieve raw storageSize.probe_compacted_ledger task (io.kestra.plugin.core.http.Request) queries GET /internalStats for compactedLedger.size -- the real byte size of the compacted ledger. (The /stats endpoint's compaction object only exposes event counts and timestamps, never a byte size.)probe_topic_reader task (io.kestra.plugin.pulsar.Reader) samples recent messages over the native binary protocol (pulsar://<broker>:6650) using a non-durable reader to verify broker connectivity without advancing consumer group cursors.evaluate_compaction_health task (io.kestra.plugin.core.output.OutputValues) computes the uncompacted ledger volume (storageSize minus compactedLedger.size) and classifies topic health into HEALTHY, COMPACTION_IN_PROGRESS, COMPACTION_FAILED, or COMPACTION_OVERDUE.route_by_compaction_state task (io.kestra.plugin.core.flow.Switch) branches on the evaluated status:HEALTHY: Emits an audit log confirming ledger storage is within acceptable thresholds.COMPACTION_IN_PROGRESS: Logs an advisory note that background compaction is currently active, skipping redundant commands.COMPACTION_FAILED: Alerts the streaming operations team on Slack with diagnostic fields.COMPACTION_OVERDUE: Alerts Slack with uncompacted backlog volume. If auto_remediate is true and dry_run is false, it triggers on-demand topic compaction via PUT /compaction, publishes a checkpoint canary record via io.kestra.plugin.pulsar.Produce, and logs the trigger outcome.errors handler, notifying the on-call team.pulsar-admin CLI sessions.io.kestra.plugin.pulsar.Reader.auto_remediate: false and dry_run: true so operators can inspect diagnostic reports before enabling automatic compaction triggers.io.kestra.plugin.pulsar.Produce to establish deterministic boundaries for compaction cycles.Apache Pulsar provides topic compaction mechanics, but does not provide an external health orchestrator to monitor compaction failures, classify ledger growth trends, or coordinate multi-step remediation with notification pipelines. Ad-hoc bash cron scripts lack centralized execution tracking, credential masking, and failure alerting.
Kestra bridges this gap by declaring the inspection, classification, remediation, and alerting sequence in version-controlled YAML. Kestra manages JWT tokens securely via the secret store, isolates network calls with declarative retries, and maintains a complete execution history for compliance and operational reviews.
apachepulsar/pulsar:latest Docker image.PULSAR_AUTH_TOKEN: JWT Bearer token authorized to perform administrative operations and read/produce messages on the target Pulsar tenant.PULSAR_GUARD_WEBHOOK_KEY: Secret authentication key securing the external Kestra webhook trigger.SLACK_WEBHOOK_URL: HTTPS endpoint for delivering structured incident alerts and remediation summaries to Slack.PULSAR_AUTH_TOKEN, PULSAR_GUARD_WEBHOOK_KEY, and SLACK_WEBHOOK_URL.docker run -d --name pulsar-standalone -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:3.2.0 bin/pulsar standalone
dry_run: true and auto_remediate: false. Verify that probe_compaction_status and probe_topic_stats fetch live metrics without errors.disabled: false on daily_maintenance_schedule once endpoints and thresholds are confirmed.io.kestra.plugin.core.http.Request task to export uncompacted backlog bytes and compaction duration to Prometheus Pushgateway.compactionThreshold via POST /admin/v2/namespaces/{tenant}/{namespace}/compactionThreshold based on daily traffic volatility.flows/pulsar-dead-letter-topic-sentinel.yaml to create an end-to-end Pulsar queue hygiene suite.COMPACTION_FAILED alerts to PagerDuty via io.kestra.plugin.pagerduty.notifications.PagerDutyAlert for 24/7 on-call paging.