Schedule icon
Webhook icon
Query icon
OutputValues icon
Switch icon
Log icon
If icon
Execute icon

ScyllaDB CDC Log Retention & Consumer Lag Headroom Guard

Audit ScyllaDB CDC table retention options and consumer lag headroom, classify expiration risk, and safely extend TTL to prevent data loss.

Categories
Data

ScyllaDB Change Data Capture (CDC) streams table mutations to a dedicated log table (<keyspace>.<table>_scylla_cdc_log) governed by a Time-To-Live (TTL) configuration. In high-throughput architectures, if downstream CDC consumers (such as Kafka Connect, Debezium, or Apache Flink) stall or lag beyond the table's configured TTL window, ScyllaDB background compaction permanently drops unconsumed records, resulting in silent data loss and replica desynchronization. Conversely, omitting a TTL or setting it to zero causes unbounded SSTable growth that can exhaust node disk capacity.

This blueprint implements a Day 2 operations guard for ScyllaDB CDC pipelines. On a recurring schedule or event trigger, it queries the local coordinator and cluster schema catalogs via io.kestra.plugin.scylladb.Query, inspects active stream generations, compares consumer checkpoints against the configured TTL window, classifies operational risk into distinct tiers (CRITICAL_LAG_RISK, UNBOUNDED_RETENTION_DRIFT, CDC_NOT_ENABLED, or HEALTHY), and conditionally executes safe schema alterations via io.kestra.plugin.scylladb.Execute to extend retention or enforce storage bounds.

How it works

  1. Trigger: A recurring schedule (io.kestra.plugin.core.trigger.Schedule) audits ScyllaDB CDC tables every 15 minutes, with an optional authenticated Webhook trigger (io.kestra.plugin.core.trigger.Webhook) for on-demand pre-maintenance checks.
  2. Coordinator probe: read_cluster_coordinator (io.kestra.plugin.scylladb.Query) queries system.local to verify driver connectivity, obtain the coordinator cluster time, and record datacenter topology.
  3. Metadata inspection: read_cdc_table_metadata (io.kestra.plugin.scylladb.Query) queries system_schema.tables with parameterized values to extract the table's live cdc configuration map, verifying whether CDC is enabled and reading the configured ttl.
  4. Generation tracking: read_cdc_generations (io.kestra.plugin.scylladb.Query) reads stream generation records from system_distributed.cdc_generations to ensure stream topology is valid.
  5. Headroom calculation: evaluate_cdc_operational_health (io.kestra.plugin.core.output.OutputValues) computes consumer lag by comparing the coordinator timestamp against the downstream consumer checkpoint (passed as input or retrieved from Kestra KV store), calculating remaining headroom seconds.
  6. Multi-tier classification: The flow evaluates whether remaining headroom has fallen below the critical percentage threshold (inputs.critical_headroom_threshold_pct), whether retention is unbounded, or whether CDC is healthy.
  7. Guarded routing: route_operational_action (io.kestra.plugin.core.flow.Switch) branches on the classification:
    • On CRITICAL_LAG_RISK: Logs an alert warning of imminent mutation drops. If inputs.auto_remediate is enabled, executes emergency_extend_cdc_ttl via io.kestra.plugin.scylladb.Execute (ALTER TABLE ... WITH cdc = {'enabled': 'true', 'ttl': ...}) to temporarily widen the retention window while downstream consumers catch up.
    • On UNBOUNDED_RETENTION_DRIFT: Logs a storage bloat warning. If inputs.auto_remediate is enabled, executes enforce_safe_cdc_ttl to apply a bounded 24-hour TTL ceiling.
    • On CDC_NOT_ENABLED: Emits an advisory notice that CDC is inactive on the target table.
    • On HEALTHY: Emits a structured telemetry audit log containing exact lag and headroom metrics.

What you get

  • Automated detection of consumer lag approaching TTL boundaries before SSTable compaction permanently purges unconsumed data.
  • Safe, automated schema remediation via ALTER TABLE to dynamically extend retention during consumer outages.
  • Identification and remediation of unbounded CDC tables (missing or zero TTL) to prevent cluster disk exhaustion.
  • Lineage-tracked execution history showing consumer lag trends and schema change audit logs in the Kestra UI.

Who it's for

  • Data Platform Engineers operating real-time CDC ingestion pipelines from ScyllaDB into Apache Kafka or event brokers.
  • Distributed Database Administrators (DBAs) responsible for ScyllaDB cluster disk capacity and SSTable compaction health.
  • Site Reliability Engineers (SREs) monitoring data synchronization SLAs and streaming consumer headroom.

Why orchestrate this with Kestra

Monitoring ScyllaDB CDC through standalone shell scripts or cron jobs leaves operational blind spots: scripts lack atomic secret masking, centralized audit histories, and reliable conditional branching. If an automated script fails while altering schema options, there is no execution retry or failure notification.

Orchestrating this workflow with Kestra provides declarative YAML management, native connection pooling via io.kestra.plugin.scylladb, integrated secret handling ({{ secret(...) }}), and structured branch routing via Switch and If. Kestra preserves a complete historical timeline of every lag measurement and schema alteration, ensuring team visibility and compliance traceability without maintaining bespoke watcher daemons.

Prerequisites

  • A running ScyllaDB cluster (version 5.0+) with Change Data Capture enabled (cdc_mode: true in scylla.yaml).
  • Network reachability from Kestra worker nodes to ScyllaDB native CQL port (default 9042).
  • A ScyllaDB user role with SELECT permissions on system_schema.tables, system.local, and system_distributed.cdc_generations, and ALTER permission on the target monitored table if auto-remediation is enabled.

Secrets

  • SCYLLADB_HOST: Hostname or IP address of a ScyllaDB node (e.g. scylla.internal or localhost).
  • SCYLLADB_PORT: Native CQL binary protocol port (typically 9042).
  • SCYLLADB_LOCAL_DATACENTER: Local datacenter name recognized by the ScyllaDB driver (e.g. datacenter1).
  • SCYLLADB_USERNAME: Username for database authentication.
  • SCYLLADB_PASSWORD: Password for database authentication.
  • CDC_HEALTH_WEBHOOK_KEY: Secret authentication key for the inbound Webhook trigger.

Quick start

  1. Start a local ScyllaDB container with experimental and CDC features enabled:
    docker run -d --name scylla-cdc-node -p 9042:9042 scylladb/scylla:latest --smp 1 --memory 1G --overprovisioned 1 --experimental 1
    
  2. Wait for the node to initialize, then create a test keyspace and CDC-enabled table using cqlsh:
    CREATE KEYSPACE IF NOT EXISTS analytics WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
    CREATE TABLE IF NOT EXISTS analytics.user_events (
        user_id uuid,
        event_time timestamp,
        event_type text,
        payload text,
        PRIMARY KEY (user_id, event_time)
    ) WITH cdc = {'enabled': 'true', 'ttl': 86400};
    
  3. Configure the required secrets (SCYLLADB_HOST, SCYLLADB_PORT, SCYLLADB_LOCAL_DATACENTER, SCYLLADB_USERNAME, SCYLLADB_PASSWORD, CDC_HEALTH_WEBHOOK_KEY) in your Kestra namespace.
  4. Import this blueprint into Kestra, execute the flow with default inputs, and review the evaluated headroom in the execution logs.
  5. Enable the schedule trigger to maintain automated 15-minute operational monitoring.

How to extend

  • Add alert channels: integrate io.kestra.plugin.slack.notifications.SlackIncomingWebhook or PagerDuty inside the CRITICAL_LAG_RISK branch to immediately notify on-call engineers.
  • Downstream consumer scaling: connect an io.kestra.plugin.kubernetes.kubectl.Patch task in the remediation branch to automatically scale up CDC consumer pod replicas when consumer lag surges.
  • Multi-table audit: wrap the table check in an io.kestra.plugin.core.flow.Loop loop to scan all CDC-enabled tables across multiple keyspaces in a single audit run.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.