Pulsar Topic Activity Sentinel
Non-destructively sample a Pulsar topic on a schedule with a non-durable reader and page Slack when it goes silent, floods, or the broker is unreachable.
Audit ScyllaDB CDC table retention options and consumer lag headroom, classify expiration risk, and safely extend TTL to prevent data loss.
ScyllaDB Change Data Capture (CDC) streams table mutations to a dedicated log table (<keyspace>.<table>_scylla_cdc_log) governed by a Time-To-Live (TTL) configuration. In high-throughput architectures, if downstream CDC consumers (such as Kafka Connect, Debezium, or Apache Flink) stall or lag beyond the table's configured TTL window, ScyllaDB background compaction permanently drops unconsumed records, resulting in silent data loss and replica desynchronization. Conversely, omitting a TTL or setting it to zero causes unbounded SSTable growth that can exhaust node disk capacity.
This blueprint implements a Day 2 operations guard for ScyllaDB CDC pipelines. On a recurring schedule or event trigger, it queries the local coordinator and cluster schema catalogs via io.kestra.plugin.scylladb.Query, inspects active stream generations, compares consumer checkpoints against the configured TTL window, classifies operational risk into distinct tiers (CRITICAL_LAG_RISK, UNBOUNDED_RETENTION_DRIFT, CDC_NOT_ENABLED, or HEALTHY), and conditionally executes safe schema alterations via io.kestra.plugin.scylladb.Execute to extend retention or enforce storage bounds.
io.kestra.plugin.core.trigger.Schedule) audits ScyllaDB CDC tables every 15 minutes, with an optional authenticated Webhook trigger (io.kestra.plugin.core.trigger.Webhook) for on-demand pre-maintenance checks.read_cluster_coordinator (io.kestra.plugin.scylladb.Query) queries system.local to verify driver connectivity, obtain the coordinator cluster time, and record datacenter topology.read_cdc_table_metadata (io.kestra.plugin.scylladb.Query) queries system_schema.tables with parameterized values to extract the table's live cdc configuration map, verifying whether CDC is enabled and reading the configured ttl.read_cdc_generations (io.kestra.plugin.scylladb.Query) reads stream generation records from system_distributed.cdc_generations to ensure stream topology is valid.evaluate_cdc_operational_health (io.kestra.plugin.core.output.OutputValues) computes consumer lag by comparing the coordinator timestamp against the downstream consumer checkpoint (passed as input or retrieved from Kestra KV store), calculating remaining headroom seconds.inputs.critical_headroom_threshold_pct), whether retention is unbounded, or whether CDC is healthy.route_operational_action (io.kestra.plugin.core.flow.Switch) branches on the classification:CRITICAL_LAG_RISK: Logs an alert warning of imminent mutation drops. If inputs.auto_remediate is enabled, executes emergency_extend_cdc_ttl via io.kestra.plugin.scylladb.Execute (ALTER TABLE ... WITH cdc = {'enabled': 'true', 'ttl': ...}) to temporarily widen the retention window while downstream consumers catch up.UNBOUNDED_RETENTION_DRIFT: Logs a storage bloat warning. If inputs.auto_remediate is enabled, executes enforce_safe_cdc_ttl to apply a bounded 24-hour TTL ceiling.CDC_NOT_ENABLED: Emits an advisory notice that CDC is inactive on the target table.HEALTHY: Emits a structured telemetry audit log containing exact lag and headroom metrics.ALTER TABLE to dynamically extend retention during consumer outages.Monitoring ScyllaDB CDC through standalone shell scripts or cron jobs leaves operational blind spots: scripts lack atomic secret masking, centralized audit histories, and reliable conditional branching. If an automated script fails while altering schema options, there is no execution retry or failure notification.
Orchestrating this workflow with Kestra provides declarative YAML management, native connection pooling via io.kestra.plugin.scylladb, integrated secret handling ({{ secret(...) }}), and structured branch routing via Switch and If. Kestra preserves a complete historical timeline of every lag measurement and schema alteration, ensuring team visibility and compliance traceability without maintaining bespoke watcher daemons.
cdc_mode: true in scylla.yaml).9042).SELECT permissions on system_schema.tables, system.local, and system_distributed.cdc_generations, and ALTER permission on the target monitored table if auto-remediation is enabled.SCYLLADB_HOST: Hostname or IP address of a ScyllaDB node (e.g. scylla.internal or localhost).SCYLLADB_PORT: Native CQL binary protocol port (typically 9042).SCYLLADB_LOCAL_DATACENTER: Local datacenter name recognized by the ScyllaDB driver (e.g. datacenter1).SCYLLADB_USERNAME: Username for database authentication.SCYLLADB_PASSWORD: Password for database authentication.CDC_HEALTH_WEBHOOK_KEY: Secret authentication key for the inbound Webhook trigger.docker run -d --name scylla-cdc-node -p 9042:9042 scylladb/scylla:latest --smp 1 --memory 1G --overprovisioned 1 --experimental 1
cqlsh:CREATE KEYSPACE IF NOT EXISTS analytics WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
CREATE TABLE IF NOT EXISTS analytics.user_events (
user_id uuid,
event_time timestamp,
event_type text,
payload text,
PRIMARY KEY (user_id, event_time)
) WITH cdc = {'enabled': 'true', 'ttl': 86400};
SCYLLADB_HOST, SCYLLADB_PORT, SCYLLADB_LOCAL_DATACENTER, SCYLLADB_USERNAME, SCYLLADB_PASSWORD, CDC_HEALTH_WEBHOOK_KEY) in your Kestra namespace.io.kestra.plugin.slack.notifications.SlackIncomingWebhook or PagerDuty inside the CRITICAL_LAG_RISK branch to immediately notify on-call engineers.io.kestra.plugin.kubernetes.kubectl.Patch task in the remediation branch to automatically scale up CDC consumer pod replicas when consumer lag surges.io.kestra.plugin.core.flow.Loop loop to scan all CDC-enabled tables across multiple keyspaces in a single audit run.