ScyllaDB gc_grace_seconds Floor Guard with Kestra

id: scylladb-gc-grace-seconds-floor-guard
namespace: company.team

description: >-
  Find ScyllaDB base tables in the keyspaces you name whose gc_grace_seconds
  sits below a safe floor, with gc_grace_seconds = 0 the worst case because it
  lets tombstones for deleted rows be purged before a repair can carry the
  deletion to every replica, so a down replica can bring the deleted row back
  after it rejoins. This is zombie data. The guard reads system_schema.tables
  for those keyspaces, flags every base table under the floor and alerts Slack.
  When you allow it, it raises the setting with ALTER TABLE one table at a time,
  then reads the schema again and fails the run if any flagged table still reads
  below the floor. gc_grace_seconds is a schema setting, so the re-read confirms
  the setting changed. It does not run a repair or reconcile data already on
  disk.

inputs:
  - id: scylla_host
    type: STRING
    defaults: "scylla"
    displayName: ScyllaDB Host
    description: Hostname or IP of one ScyllaDB contact point reachable from the
      Kestra Worker.

  - id: scylla_port
    type: INT
    defaults: 9042
    displayName: ScyllaDB CQL Port
    description: Native CQL port the contact point listens on.

  - id: scylla_datacenter
    type: STRING
    defaults: "datacenter1"
    displayName: Local Datacenter
    description: Datacenter the driver treats as local for its load balancing
      policy. A default ScyllaDB install uses datacenter1.

  - id: target_keyspaces
    type: ARRAY
    itemType: STRING
    displayName: Target keyspaces
    description: Keyspaces whose base tables delete rows and so need a non-zero
      gc_grace_seconds, for example event, audit or time series keyspaces.
      System keyspaces are never scanned.
    defaults:
      - events

  - id: gc_grace_floor_seconds
    type: INT
    displayName: gc_grace_seconds floor (seconds)
    description: Lowest gc_grace_seconds a base table may carry. A table below this
      is flagged. Remediation sets it to exactly this value. Keep it at or above
      your longest repair interval. The ScyllaDB default is 864000 (ten days).
      3600 is one hour.
    defaults: 3600

  - id: auto_remediate
    type: BOOL
    displayName: Auto remediate
    defaults: false

  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: Must be false, together with auto_remediate true, for any ALTER
      TABLE to run.
    defaults: true

variables:
  scan_cql: >-
    SELECT keyspace_name, table_name, gc_grace_seconds FROM system_schema.tables
    WHERE keyspace_name IN ({{ inputs.target_keyspaces | jq('if length == 0 then
    "\u0027\u0027" else map("\u0027" + gsub("\u0027"; "\u0027\u0027") +
    "\u0027") | join(",") end') | first }})
  flag_count_jq: >-
    [.[] | select(((.gc_grace_seconds // 0) | tonumber) < {{
    inputs.gc_grace_floor_seconds }})] | length
  flag_names_jq: >-
    [.[] | select(((.gc_grace_seconds // 0) | tonumber) < {{
    inputs.gc_grace_floor_seconds }}) | .keyspace_name + "." + .table_name + "
    (gc_grace_seconds=" + ((.gc_grace_seconds // 0) | tostring) + ")"] | join(",
    ")
  fix_jq: >-
    [.[] | select(((.gc_grace_seconds // 0) | tonumber) < {{
    inputs.gc_grace_floor_seconds }}) | "ALTER TABLE \"" + (.keyspace_name |
    gsub("\"";"\"\"")) + "\".\"" + (.table_name | gsub("\"";"\"\"")) + "\" WITH
    gc_grace_seconds = {{ inputs.gc_grace_floor_seconds }}"]

tasks:
  - id: scan_tables
    type: io.kestra.plugin.cassandra.standard.Query
    description: Read only. Lists base tables and their gc_grace_seconds for the
      target keyspaces from system_schema.tables, restricted on the
      keyspace_name partition key so no ALLOW FILTERING and no system scan is
      needed.
    session:
      endpoints:
        - hostname: "{{ inputs.scylla_host }}"
          port: "{{ inputs.scylla_port }}"
      localDatacenter: "{{ inputs.scylla_datacenter }}"
    cql: "{{ render(vars.scan_cql) }}"
    fetchType: FETCH

  - id: flagged
    type: io.kestra.plugin.core.output.OutputValues
    description: Build the ALTER statements and a readable name list from the base
      tables below the floor.
    values:
      fix_sql: "{{ outputs.scan_tables.rows | jq(render(vars.fix_jq)) | first | toJson
        }}"
      names: "{{ outputs.scan_tables.rows | jq(render(vars.flag_names_jq)) | first }}"

  - id: summary
    type: io.kestra.plugin.core.log.Log
    message: >-
      gc_grace floor guard on {{ inputs.scylla_host }}: scanned {{
      outputs.scan_tables.size }} base tables in {{ inputs.target_keyspaces |
      jq('join(", ")') | first }}, {{ outputs.scan_tables.rows |
      jq(render(vars.flag_count_jq)) | first }} below the floor of {{
      inputs.gc_grace_floor_seconds }}s.

  - id: route
    type: io.kestra.plugin.core.flow.If
    condition: "{{ (outputs.scan_tables.rows | jq(render(vars.flag_count_jq)) |
      first) > 0 }}"
    then:
      - id: alert
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {{ {'text': 'ScyllaDB ' ~ inputs.scylla_host ~ ': base tables below the gc_grace_seconds floor of ' ~ inputs.gc_grace_floor_seconds ~ 's, so deleted rows risk resurrecting after a repair: ' ~ outputs.flagged.values.names ~ '. ' ~ (inputs.auto_remediate and not inputs.dry_run ? 'Raising gc_grace_seconds to ' ~ inputs.gc_grace_floor_seconds ~ 's now.' : 'No change applied (auto_remediate=' ~ inputs.auto_remediate ~ ' dry_run=' ~ inputs.dry_run ~ ').')} | toJson }}

      - id: remediate_gate
        type: io.kestra.plugin.core.flow.If
        condition: "{{ inputs.auto_remediate and not inputs.dry_run }}"
        then:
          - id: remediate
            type: io.kestra.plugin.core.flow.Loop
            description: Sequential remediation on purpose. Concurrent schema changes race
              ScyllaDB schema agreement and can leave nodes disagreeing on the
              schema.
            concurrencyLimit: 1
            values: "{{ outputs.flagged.values.fix_sql }}"
            tasks:
              - id: apply_gc
                type: io.kestra.plugin.cassandra.standard.Query
                description: One ALTER TABLE ... WITH gc_grace_seconds per flagged base table,
                  run through the Cassandra-compatible CQL plugin.
                allowFailure: true
                session:
                  endpoints:
                    - hostname: "{{ inputs.scylla_host }}"
                      port: "{{ inputs.scylla_port }}"
                  localDatacenter: "{{ inputs.scylla_datacenter }}"
                cql: "{{ item.value }}"

          - id: settle
            type: io.kestra.plugin.core.flow.Sleep
            description: Give the cluster a moment to reach schema agreement before the
              re-read, so a slow propagating ALTER does not read back as a false
              failure.
            duration: PT5S

          - id: reverify
            type: io.kestra.plugin.cassandra.standard.Query
            description: Read system_schema.tables again for the same keyspaces to confirm
              the floor landed in the schema.
            session:
              endpoints:
                - hostname: "{{ inputs.scylla_host }}"
                  port: "{{ inputs.scylla_port }}"
              localDatacenter: "{{ inputs.scylla_datacenter }}"
            cql: "{{ render(vars.scan_cql) }}"
            fetchType: FETCH

          - id: verify_result
            type: io.kestra.plugin.core.flow.If
            condition: "{{ (outputs.reverify.rows | jq(render(vars.flag_count_jq)) | first)
              > 0 }}"
            then:
              - id: remediation_failed
                type: io.kestra.plugin.core.execution.Fail
                errorMessage: "Base tables still read below the gc_grace_seconds floor after
                  ALTER TABLE: {{ outputs.reverify.rows |
                  jq(render(vars.flag_names_jq)) | first }}. The connecting role
                  needs ALTER on these tables."
            else:
              - id: remediated
                type: io.kestra.plugin.core.log.Log
                message: "Set and verified gc_grace_seconds = {{ inputs.gc_grace_floor_seconds
                  }}s on {{ outputs.flagged.values.names }}. This changes the
                  schema setting only. It does not run a repair or reconcile
                  rows already on disk."
    else:
      - id: clean
        type: io.kestra.plugin.core.log.Log
        message: "Every base table in {{ inputs.target_keyspaces | jq('join(\", \")') |
          first }} carries a gc_grace_seconds at or above the floor of {{
          inputs.gc_grace_floor_seconds }}s."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Alert Slack when the guard itself fails, for example an unreachable
      cluster or a role without ALTER rights, so a broken run never reads as a
      clean one.
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {'text': 'ScyllaDB gc_grace floor guard failed on ' ~ inputs.scylla_host ~ ' (execution ' ~ execution.id ~ '). The gc_grace posture is now unknown. Check the task logs.'} | toJson }}

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    description: Shipped disabled and always in safe mode (auto_remediate false,
      dry_run true) regardless of this flow's own defaults.
    cron: "25 4 * * *"
    disabled: true
    inputs:
      auto_remediate: false
      dry_run: true

A ScyllaDB table deletes a row by writing a tombstone, a marker that says the row is gone. gc_grace_seconds is how long that tombstone is kept before compaction may purge it. The window has to outlast the time a repair needs to carry the deletion to every replica. Set it too low and the worst case is 0, so a replica that was down during the delete can miss the tombstone. After the tombstone is purged that replica hands the old row back when it rejoins. The deleted row returns. People call it zombie data. Nothing in the cluster warns you, because a low or zero value is legal and is sometimes chosen on purpose for a table that never deletes.

This blueprint reads system_schema.tables for the keyspaces you name as delete-heavy, flags every base table whose gc_grace_seconds is below a floor you set and alerts Slack. When you allow it, it raises the setting with ALTER TABLE ... WITH gc_grace_seconds, one table at a time, then reads the schema again and fails the run if any flagged table still reads below the floor. It reaches ScyllaDB over the bundled Cassandra plugin (io.kestra.plugin.cassandra.standard.Query), because ScyllaDB speaks the Cassandra CQL wire protocol, so no extra plugin install is needed.

This blueprint was created by zkasuran.

Safety

gc_grace_seconds is a schema setting. Raising it changes how long future tombstones are kept. It does not run a repair. It does not reconcile replicas or undo a resurrection that already happened. The re-verify step reads system_schema.tables again and confirms the setting now sits at or above the floor. That is a check on the schema, not proof that any data was repaired. To close the window for good you still run a full repair inside gc_grace_seconds on a schedule, which this guard deliberately does not do.

The remediation loop runs one ALTER TABLE at a time (concurrencyLimit: 1). Concurrent schema changes race ScyllaDB schema agreement and can leave nodes disagreeing on the schema. Both write gates (auto_remediate true with dry_run false) must be set before any ALTER TABLE runs.

How it works

scan_tables (io.kestra.plugin.cassandra.standard.Query, read only) runs SELECT keyspace_name, table_name, gc_grace_seconds FROM system_schema.tables WHERE keyspace_name IN (...) for the target_keyspaces you pass. The restriction is on the keyspace_name partition key, so the read needs no ALLOW FILTERING and never touches a system keyspace. system_schema.tables lists base tables only, so views and indexes stay out of scope.
flagged (io.kestra.plugin.core.output.OutputValues) keeps the rows below the floor and builds one ALTER TABLE statement per table plus a readable name list. summary logs the counts.
route (io.kestra.plugin.core.flow.If) branches on whether any table is below the floor:
- clean: clean logs that every base table in the target keyspaces is at or above the floor.
- flagged: alert posts the table list to Slack.
  - remediate_gate runs only when auto_remediate is true with dry_run false.
  - remediate (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) runs each ALTER TABLE ... WITH gc_grace_seconds through io.kestra.plugin.cassandra.standard.Query in turn.
  - settle waits a few seconds for schema agreement.
  - reverify re-reads system_schema.tables. verify_result fails the run through io.kestra.plugin.core.execution.Fail if any flagged table still reads below the floor.
The errors block alerts Slack when the flow itself fails, for example an unreachable cluster or a role without ALTER rights.
The daily Schedule trigger ships disabled: true and always forces safe mode.

What you get

A standing answer to which base tables in your delete-heavy keyspaces sit below a safe gc_grace_seconds, read from ScyllaDB's own schema rather than guessed.
A floor raised only behind two gates, one table at a time, with a re-read of the schema as proof.
A Slack page the moment a table drops below the floor, plus a separate failure channel when the guard cannot run.
A clear line between a schema fix (what this does) and a repair (what it does not).

Who it's for

Platform and database reliability teams running self-managed ScyllaDB who treat silent data resurrection as an outage waiting to happen.
Teams onboarding new keyspaces where a CREATE TABLE with a low or zero gc_grace_seconds is an easy miss.
Anyone evaluating the Cassandra plugin against ScyllaDB who wants a worked system_schema.tables audit that also drives ALTER TABLE.

Why orchestrate this with Kestra

A one-off DESCRIBE in cqlsh answers the question for one table at one moment and fixes nothing. Kestra reads the whole set on a schedule, turns it into a Slack page and an execution history, applies the fix only behind auto_remediate with dry_run, one statement at a time, then verifies the result by reading the schema back. The guard becomes a recurring check instead of a thing someone remembers to run.

Tested end to end

On Kestra 2.0.5 against ScyllaDB (scylladb/scylla) on the same Docker network, with a local HTTP sink standing in for Slack. Seeded schema:

keyspace metrics_clean: table rollup_daily left at the default gc_grace_seconds = 864000;
keyspace metrics: table readings created WITH gc_grace_seconds = 60;
keyspace events: table audit_log created WITH gc_grace_seconds = 0.

Run	Inputs	Result
1	`target_keyspaces: [metrics_clean]`, floor 3600	Clean. Its one base table reads 864000. No alert
2	`target_keyspaces: [events, metrics]`, floor 3600	2 base tables flagged (`events.audit_log` at 0, `metrics.readings` at 60). Alert sent. Schema still reads 0 and 60
3	run 2 keyspaces, `auto_remediate: true`, `dry_run: true`	Same report. cqlsh confirms both unchanged. No ALTER ran
4	run 2 keyspaces, `auto_remediate: true`, `dry_run: false`	Both `ALTER TABLE` ran one at a time. `reverify` found 0 below the floor. cqlsh confirms both now read 3600
5	`scylla_host` pointed at an unreachable name	`scan_tables` failed, the `errors` block posted to the sink, execution FAILED

Before and after values from the cluster's own CLI are in the PR description.

Prerequisites

A reachable ScyllaDB cluster with its native CQL port open to the Kestra Worker. For the audit, a role that can read system_schema.tables. For remediation, a role with ALTER on the target tables.
The bundled Cassandra plugin (io.kestra.plugin.cassandra.standard.Query). ScyllaDB speaks the Cassandra CQL wire protocol, so this plugin ships with Kestra and needs no separate install.
Local testing: docker run -d --name scylla --network <kestra-network> scylladb/scylla --smp 1 --memory 1G --overprovisioned 1 --developer-mode 1, then create a keyspace and a table with a low gc_grace_seconds. Allow a minute or two for the single node to accept CQL.

Secrets

SLACK_WEBHOOK_URL: Slack incoming webhook used by alert and the errors block.
This flow connects without a username or password, which matches a local developer-mode ScyllaDB. For an authenticated cluster, add a username plus a password: {{ secret('SCYLLA_PASSWORD') }} to each task's session rather than hardcoding them. The driver needs both or neither.
In Kestra OSS, provide a secret as a base64-encoded environment variable prefixed SECRET_, for example SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
`scylla_host`	`scylla`	Contact point hostname or IP
`scylla_port`	`9042`	Native CQL port
`scylla_datacenter`	`datacenter1`	Local datacenter for the driver
`target_keyspaces`	`[events]`	Keyspaces whose base tables delete rows
`gc_grace_floor_seconds`	`3600`	Lowest gc_grace_seconds a base table may carry
`auto_remediate`	`false`	Allow `ALTER TABLE`
`dry_run`	`true`	Must be `false` for a statement to run

Quick start

Add the SLACK_WEBHOOK_URL secret. The Cassandra plugin that reaches ScyllaDB ships with Kestra, so there is nothing extra to install.
Import the flow and set target_keyspaces to the keyspaces whose tables delete rows.
Run it once with the defaults and read which base tables sit below the floor.
Agree a gc_grace_floor_seconds with the owners, then run with auto_remediate: true and dry_run: false.
Enable the daily trigger, which always runs in safe mode.

How to extend

Add a Pause before the loop for a human approval step on production clusters.
Set a different floor per keyspace from a map, for example a longer window for audit tables.
Also flag tables whose default_time_to_live is 0 in keyspaces meant to expire data.
Pair with a separate flow that runs a scheduled repair inside gc_grace_seconds, which is what actually closes the resurrection window.

ScyllaDB gc_grace_seconds Floor Guard

Safety

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

ScyllaDB gc_grace_seconds Floor Guard

Safety

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Kafka ACL Baseline Restore Guard

Archive ownerless failed Pod diagnostics before cleanup

Kafka Client Quota Guard

Kafka Consumer Group Dead-Letter Rebalance and Offset Rewind Guard

Kafka Consumer Lag Sentinel

Kafka ACL Overly Broad Grant Audit

More Data Blueprints

Incremental Docs-to-Chroma RAG Sync with Deletions and a Retrieval Smoke Test

Build a Chat-With-Your-Data System Using Elasticsearch and OpenAI (RAG)

Watch Cited Papers for Retractions, Expressions of Concern and Corrections via Crossref

Citus Colocation Violation Guard

Citus Inactive Worker Reactivate Guard

Score change requests on risk and build the CAB agenda, pre-approving the low-risk ones

New to Kestra?