Backfill Camunda Messages Lost Between an Outbox and a Catch Event
Find overdue Camunda message subscriptions with matching PostgreSQL outbox rows, then safely republish and prove the instance moved.
Flag Cassandra base tables with compression disabled, alert Slack, turn on LZ4 with ALTER TABLE behind an approval and re-read the schema.
CREATE TABLE ... WITH compression = {'enabled': false} is one line in a schema file that nobody revisits after the table exists. It tells Cassandra to write every SSTable for this table raw. On a large table that wastes disk. It also wastes read IO on every read, because each lookup pulls more bytes off disk than a compressed chunk would. The default is LZ4 compression precisely because the trade is almost always worth it, so a table sitting at disabled is usually a mistake rather than a decision.
This blueprint reads system_schema.tables for the keyspaces you name, flags every base table whose compression map carries no compressor class and alerts Slack. When you allow it, it turns on LZ4 with ALTER TABLE ... WITH compression, one table at a time, behind a human approval, then reads the schema again and fails the run if any flagged table still has no compressor.
This blueprint was created by zkasuran.
ALTER TABLE ... WITH compression changes the setting straight away, so every SSTable written after the change is compressed. It does NOT rewrite the data already on disk. Existing SSTables are recompressed only as future compaction rewrites them, so disk is reclaimed over time, not at the moment of the ALTER. The re-verify step is a schema check: it confirms system_schema.tables now shows a compressor class on each flagged table, which proves the setting landed, not that every byte on disk is already compressed. To rewrite existing data at once run nodetool upgradesstables -a on each node, which this guard deliberately does not do.
The remediation loop runs one ALTER TABLE at a time (concurrencyLimit: 1). Concurrent schema changes race Cassandra's schema agreement and can leave nodes disagreeing on the schema. Both write gates (auto_remediate true with dry_run false) must be set before any ALTER TABLE runs. A human must also approve the Pause.
scan_tables (io.kestra.plugin.cassandra.standard.Query, read only) runs SELECT keyspace_name, table_name, compression FROM system_schema.tables WHERE keyspace_name IN (...) for the target_keyspaces you pass. Any name starting with system is dropped from the list first, so the built in keyspaces are never in scope. The restriction is on the keyspace_name partition key, so the read needs no ALLOW FILTERING. system_schema.tables lists base tables only, so views and indexes are out of scope by construction.flagged (io.kestra.plugin.core.output.OutputValues) keeps the rows whose compression map has no class key, which is how a disabled table reads back, then builds one ALTER TABLE per table plus a readable name list. summary logs the counts.route (io.kestra.plugin.core.flow.If) branches on whether any table has compression disabled:clean logs that every base table in the target keyspaces carries a compressor class.alert posts the table list to Slack.remediate_gate runs only when auto_remediate is true with dry_run false.hold (io.kestra.plugin.core.flow.Pause) waits for a person to approve the list.remediate (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) runs each ALTER TABLE ... WITH compression through io.kestra.plugin.cassandra.standard.Query in turn.settle waits a few seconds for schema agreement, then reverify re-reads system_schema.tables. verify_result fails the run through io.kestra.plugin.core.execution.Fail if any flagged table still has no compressor class.errors block alerts Slack when the flow itself fails, for example an unreachable cluster or a role without ALTER rights.daily Schedule trigger ships disabled: true and always forces safe mode.nodetool upgradesstables -a does).CREATE TABLE with compression off is an easy and expensive miss.system_schema.tables audit that also drives ALTER TABLE.A DESCRIBE TABLE in cqlsh answers this for one table at one moment and changes nothing. Kestra reads the whole set on a schedule, turns it into a Slack page and an execution history, applies the fix only behind auto_remediate with dry_run plus a human approval, one statement at a time, then verifies by reading the schema back. A table created with compression off next month is caught by the next scheduled run instead of by a disk alert.
On Kestra 2.0.5 OSS against cassandra:5 (5.0.x) on the same Docker network, with a local HTTP sink standing in for Slack. Seeded keyspace app: events and readings created WITH compression = {'enabled': false}, so both read back with no compressor class, plus profiles created WITH compression = {'class': 'LZ4Compressor'} which is left untouched to prove precision. Keyspace app_clean holds one table on the default LZ4.
| Run | Inputs | Result |
|---|---|---|
| 1 | target_keyspaces: [app_clean] |
Clean. 1 base table scanned, 0 flagged. No alert, no change |
| 2 | target_keyspaces: [app] |
3 scanned, 2 flagged (app.events, app.readings). Alert sent. profiles on LZ4 not flagged. Schema unchanged |
| 3 | run 2 plus auto_remediate: true, dry_run: true |
Same report. cqlsh confirms both still disabled. No ALTER ran, the Pause was never reached |
| 4 | run 2, auto_remediate: true, dry_run: false, resumed with no answer |
Pause reached, declined branch taken, schema untouched |
| 5 | same, approved | 2 ALTER TABLE ran one at a time, reverify found 0 disabled. cqlsh: both now carry the LZ4Compressor class, profiles still LZ4 |
| 6 | same, with app.hotpath created with compression off while the run waited for approval |
The 2 known tables were fixed, reverify found app.hotpath, the run FAILED and the errors block posted |
| 7 | target_keyspaces: [system, system_schema, app_clean] |
1 base table scanned. The system names are dropped before the query runs |
| 8 | cassandra_host pointed at an unreachable name |
scan_tables failed, the errors block posted to the sink, execution FAILED |
Before and after compression maps from system_schema.tables read with cqlsh are in the PR description.
system_schema.tables. For remediation, a role with ALTER on the target tables.io.kestra.plugin.cassandra.standard.Query), which ships with Kestra.docker run -d --name cassandra --network <kestra-network> -e MAX_HEAP_SIZE=1G -e HEAP_NEWSIZE=256M cassandra:5, then create a keyspace and a table WITH compression = {'enabled': false}. Allow a minute or two before the node accepts CQL.SLACK_WEBHOOK_URL: Slack incoming webhook used by alert and the errors block.username plus a password: {{ secret('CASSANDRA_PASSWORD') }} to each task's session rather than hardcoding them. The driver needs both or neither.SECRET_, for example SECRET_SLACK_WEBHOOK_URL.| Input | Default | Purpose |
|---|---|---|
cassandra_host |
cassandra |
Contact point hostname or IP |
cassandra_port |
9042 |
Native CQL port |
cassandra_datacenter |
datacenter1 |
Local datacenter for the driver |
target_keyspaces |
[app] |
Keyspaces to audit. Names starting with system are dropped |
auto_remediate |
false |
Allow ALTER TABLE |
dry_run |
true |
Must be false before the approval step is reached |
SLACK_WEBHOOK_URL secret. The Cassandra plugin ships with Kestra, so there is nothing extra to install.target_keyspaces to your application keyspaces.auto_remediate: true and dry_run: false and approve the Pause.daily trigger, which always runs in safe mode.nodetool upgradesstables -a per node to rewrite existing data at once rather than waiting for compaction.ZstdCompressor for cold archive tables and LZ4Compressor for hot ones.flagged.chunk_length_in_kb for the read pattern, not only tables with no compressor at all.