Flag Cassandra tables where a high compaction tombstone_threshold stops tombstone purging, alert Slack, tune the subproperties with ALTER TABLE and verify.
A delete heavy Cassandra table writes a tombstone for every deleted or expired row. Those tombstones only leave disk when compaction purges them. Two compaction subproperties decide how eagerly that happens. tombstone_threshold is the fraction of a single SSTable that must be tombstones before Cassandra will compact that one SSTable on its own to purge them. Its engine default is 0.2. Set it near 1.0 and the SSTable has to be almost all tombstones before anything triggers, so in practice single SSTable tombstone compaction never runs. unchecked_tombstone_compaction decides whether the engine still runs that compaction when it judges the run unproductive. Its default is false, which means a borderline SSTable is skipped. Put a high tombstone_threshold next to the default false and a table that deletes a lot keeps its tombstones. Reads that scan past them get slow. A partition with enough of them answers with TombstoneOverwhelmingException.
This blueprint reads system_schema.tables for the keyspaces you name, flags every table whose tombstone_threshold is above a ceiling you set while unchecked_tombstone_compaction is not true and alerts Slack. When you allow it, it rewrites only those two subproperties with ALTER TABLE ... WITH compaction, one table at a time, behind a human approval, then reads the schema again and fails the run if any flagged table still reads above the ceiling.
This blueprint was created by zkasuran.
Two honest notes, because the useful version of this guard is narrower than the scary version.
First, an absent tombstone_threshold is not a problem. A table that never set the option reads as the engine default 0.2, which is a low, purge friendly value, so this guard treats absent as safe and never flags it. The dangerous table is the one where somebody wrote a high value on purpose and moved on. That is why the detector reads the real value from the schema rather than assuming a default.
Second, raising unchecked_tombstone_compaction makes single SSTable tombstone compaction eligible more often. It does not force a tombstone to drop. Cassandra still only drops a tombstone when it is safe, which means the tombstone is older than gc_grace_seconds and no overlapping SSTable still needs it. So this flow fixes the setting that was stopping the compaction from being considered. It does not promise that every tombstone leaves disk on the next pass. The honest claim is exact and checkable from the schema: these tables carried a configuration that suppressed single SSTable tombstone compaction. Now they do not.
The precision rule is the point of this blueprint. A table's compaction map holds its strategy class (SizeTieredCompactionStrategy, LeveledCompactionStrategy, TimeWindowCompactionStrategy and so on) plus the options that strategy was tuned with. A naive ALTER TABLE ... WITH compaction = {'class': 'SizeTieredCompactionStrategy', ...} would overwrite all of that and could silently change the table's real strategy, which is a far worse change than the one it set out to make. So the guard reads the current compaction map first, merges only tombstone_threshold and unchecked_tombstone_compaction into it and writes the whole merged map back. The class and every other option are preserved exactly. The re-verify step reads system_schema.tables again and the PR description shows the compaction map before and after, including the unchanged class.
The remediation loop runs one ALTER TABLE at a time (concurrencyLimit: 1). Concurrent schema changes race Cassandra schema agreement and can leave nodes disagreeing on the schema. Both write gates (auto_remediate true with dry_run false) plus a human approval must line up before any ALTER TABLE runs.
scan_tables (io.kestra.plugin.cassandra.standard.Query, read only) runs SELECT keyspace_name, table_name, compaction FROM system_schema.tables WHERE keyspace_name IN (...) for the target_keyspaces you pass. Any name starting with system is dropped from the list first, so the built in keyspaces are never in scope. The restriction is on the keyspace_name partition key, so the read needs no ALLOW FILTERING.flagged (io.kestra.plugin.core.output.OutputValues) keeps the rows whose tombstone_threshold is above tombstone_threshold_ceiling while unchecked_tombstone_compaction is not true, merges the two subproperties into each table's existing compaction map and builds one ALTER TABLE per table plus a readable name list. summary logs the counts.route (io.kestra.plugin.core.flow.If) branches on whether any table is above the ceiling:clean logs that every table in the target keyspaces can run single SSTable tombstone compaction.alert posts the table list to Slack.remediate_gate runs only when auto_remediate is true with dry_run false.hold (io.kestra.plugin.core.flow.Pause) waits for a person to approve the list.remediate (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) runs each merged ALTER TABLE ... WITH compaction through io.kestra.plugin.cassandra.standard.Query in turn.settle waits a few seconds for schema agreement, then reverify re-reads system_schema.tables. verify_result fails the run through io.kestra.plugin.core.execution.Fail if any table still reads above the ceiling.errors block alerts Slack when the flow itself fails, for example an unreachable cluster or a role without ALTER rights.daily Schedule trigger ships disabled: true and always forces safe mode.CREATE TABLE somebody committed two years ago.class and every other option left exactly as they were, so a Leveled or TimeWindow table stays on its strategy.tombstone_threshold was copied between CREATE TABLE statements without anyone noticing.TombstoneOverwhelmingException on a read path and wants the compaction side of the fix enforced on a schedule.A DESCRIBE TABLE in cqlsh answers this for one table at one moment and changes nothing. Kestra reads the whole set on a schedule, turns it into a Slack page and an execution history, merges the fix into each table's existing compaction map only behind auto_remediate with dry_run plus a human approval, one statement at a time, then verifies by reading the schema back. A table created with a high tombstone_threshold next month is caught by the next scheduled run instead of by a read timeout.
On Kestra 2.0.5 against cassandra:5 (5.0) on the same Docker network, with a local HTTP sink standing in for Slack. Seeded keyspace app: events created WITH compaction = {'class': 'SizeTieredCompactionStrategy', 'tombstone_threshold': '0.99', 'unchecked_tombstone_compaction': 'false'}; logs_lcs created on LeveledCompactionStrategy with tombstone_threshold: '0.95' to prove the class survives the ALTER; metrics_tuned left on STCS with unchecked_tombstone_compaction: 'true' so it is never flagged; plain_default created with no tombstone options so it reads the engine default 0.2. Keyspace app_clean holds one table at the defaults.
| Run | Inputs | Result |
|---|---|---|
| 1 | target_keyspaces: [app_clean], ceiling 0.6 |
Clean. 1 table scanned, 0 flagged. No alert, no change |
| 2 | target_keyspaces: [app] |
4 scanned, 2 flagged (app.events at 0.99, app.logs_lcs at 0.95). Alert sent. metrics_tuned and plain_default not flagged. Schema unchanged |
| 3 | run 2 plus auto_remediate: true, dry_run: true |
Same report. cqlsh confirms all 4 unchanged. No ALTER ran, the Pause was never reached |
| 4 | auto_remediate: true, dry_run: false, resumed with no answer |
Pause reached, declined branch taken, schema untouched |
| 5 | same, approved | 2 ALTER TABLE ran one at a time, reverify found 0 above the ceiling. cqlsh: events now 0.2 with unchecked_tombstone_compaction true and class still SizeTieredCompactionStrategy, logs_lcs now 0.2/true with class still LeveledCompactionStrategy and sstable_size_in_mb kept |
| 6 | same, with app.hotpath created at 0.99 while the run waited for approval |
The 2 known tables were fixed, reverify found app.hotpath, the run FAILED and the errors block posted |
| 7 | target_keyspaces: [system, system_schema, app_clean] |
1 table scanned. The system names are dropped before the query runs |
| 8 | cassandra_host pointed at an unreachable name |
scan_tables failed, the errors block posted to the sink, execution FAILED |
Before and after compaction maps from cqlsh are in the PR description, including the proof that each table kept its compaction class.
system_schema.tables. For remediation, a role with ALTER on the target tables.io.kestra.plugin.cassandra.standard.Query), which ships with Kestra.docker run -d --name cassandra --network <kestra-network> -e MAX_HEAP_SIZE=1G -e HEAP_NEWSIZE=256M cassandra:5, then create a keyspace and a table WITH compaction = {'class': 'SizeTieredCompactionStrategy', 'tombstone_threshold': '0.99', 'unchecked_tombstone_compaction': 'false'}. Allow a minute or two before the node accepts CQL.SLACK_WEBHOOK_URL: Slack incoming webhook used by alert and the errors block.username plus a password: {{ secret('CASSANDRA_PASSWORD') }} to each task's session rather than hardcoding them. The driver needs both or neither.SECRET_, for example SECRET_SLACK_WEBHOOK_URL.| Input | Default | Purpose |
|---|---|---|
cassandra_host |
cassandra |
Contact point hostname or IP |
cassandra_port |
9042 |
Native CQL port |
cassandra_datacenter |
datacenter1 |
Local datacenter for the driver |
target_keyspaces |
[app] |
Keyspaces to audit. Names starting with system are dropped |
tombstone_threshold_ceiling |
0.6 |
Highest tombstone_threshold allowed before a table is flagged |
remediation_tombstone_threshold |
0.2 |
Value written by the fix. Keep it at or below the ceiling |
auto_remediate |
false |
Allow ALTER TABLE |
dry_run |
true |
Must be false before the approval step is reached |
SLACK_WEBHOOK_URL secret. The Cassandra plugin ships with Kestra, so there is nothing extra to install.target_keyspaces to your application keyspaces.tombstone_threshold.auto_remediate: true and dry_run: false and approve the Pause.daily trigger, which always runs in safe mode.tombstone_threshold, then skip them in flagged.tombstone_compaction_interval on the same tables, the minimum age before a single SSTable is eligible for a tombstone compaction.scylladb-gc-grace-seconds-floor-guard or a scheduled repair, so the retention side of the tombstone lifecycle is covered as well as the purge side.Start a critical Camunda 8 process, classify its completion HEALTHY, AT_RISK, or UNREACHABLE by its own outcome variable, alert Slack, and optionally retry.
Find Camunda 8 instances whose job retries are exhausted and whose incident is past a grace window, cancel them behind an approval gate, then prove from the engine that they are gone.
Audits Capsule operator telemetry for tenant namespace quota breaches, owner lock cordons, and proxy drops.
Monitor Cassandra 4.0+ virtual tables on schedule, detect saturated thread pools with blocked or pending tasks, and alert Slack.
Flag Cassandra base tables stored uncompressed, alert Slack, then turn on LZ4 compression with ALTER TABLE behind an approval and re-read the schema.
Extract Cassandra data with CQL, serialize it to CSV, and load it into Google BigQuery for analytics.
