Ceph Pool Replication Compliance Gate

A Ceph pool created with size: 1 or size: 2 during a rushed provisioning step looks completely normal - it accepts writes, serves reads, shows up in every list. It just has far less data redundancy than it should, and the gap usually surfaces the hard way, during an OSD failure that a properly replicated pool would have absorbed without incident. This blueprint lists every pool through the Ceph Dashboard API, flags any replicated pool below a minimum replica count you set, alerts Slack by name, and can raise the replica size back up - gated behind two independent confirmations, and only ever raising size, never lowering it.

How it works

1. periodic_replication_check (io.kestra.plugin.core.trigger.Schedule) runs every 30 minutes, always forcing auto_remediate: "false" and dry_run: "true" through its own inputs: override regardless of this flow's defaults. Shipped disabled so you can validate a manual run first.
2. list_pools (io.kestra.plugin.ceph.pools.List, fetchType: FETCH) calls GET /api/pool, returning every pool's pool_name, type, size, pg_num, and more under pools.
3. evaluate_pool_replication (io.kestra.plugin.core.flow.If) filters pools with a jq expression for type == "replicated" and size < min_replica_size, branching on whether that count is greater than zero.
   • else (HEALTHY): log_replication_healthy records a clean check.
   • then (AT_RISK): route_remediation (io.kestra.plugin.core.flow.Switch on auto_remediate) either considers remediation ("true", gated again by dry_run through check_dry_run) or only logs that remediation is disabled ("false"). When both gates allow it, raise_under_replicated_pools (io.kestra.plugin.core.flow.Loop over the same filtered list) calls io.kestra.plugin.ceph.pools.Update with size: min_replica_size on each affected pool only. alert_under_replicated_pools always fires to Slack, naming every affected pool and its current size, regardless of which remediation path ran.
4. log_audit_complete always runs last, printing the headline numbers regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable Dashboard host, bad credentials, or a failed Update call must not read as "no risk".

What you get

• A direct read of every pool's actual size from the Ceph Dashboard API, not an assumption based on when the pool was created or who created it.
• The exact pool names and current sizes in the Slack alert, so triage starts with a list, not a bare count.
• Remediation scoped to exactly size, raised to the minimum - pgNum and applicationMetadata are left untouched on every pool this flow updates, and a pool already at or above the minimum is never included in the update loop at all.
• Two independent gates (auto_remediate, dry_run) between "gap detected" and "replica size actually raised".
• A final status log every run, gap or not, so the execution history doubles as a replication-compliance trend across the whole cluster.

Who it's for

• Storage and platform teams running Ceph who need a standing check that every pool meant to be triple-replicated (or whatever your minimum is) actually is, not just the ones someone remembered to check at provisioning time.
• SREs doing periodic durability reviews who want the exact under-replicated pool list pushed to Slack instead of running ceph osd pool ls detail by hand.
• Anyone evaluating the Ceph plugin who wants a second worked example beyond cluster.GetHealth - this one exercises pools.List and pools.Update, a completely different package with its own real governance angle: replication/durability compliance, not overall cluster status.

Why orchestrate this with Kestra

ceph osd pool ls detail run by hand answers one moment in time; it does not decide a cadence, filter for the pools that actually violate a policy, gate a fix behind two independent confirmations, or notify anyone. Kestra supplies the schedule, the threshold filter as a first-class branch, a Loop that updates only what is actually out of compliance, and an execution history that shows exactly when a pool fell out of replication compliance and whether it was fixed.

Prerequisites

• A reachable Ceph cluster with the Dashboard Manager module enabled and at least one replicated pool.
• A Dashboard account with permission to read /api/pool and, if auto_remediate will ever be "true", to call PUT /api/pool/{pool_name}.
• A Slack incoming webhook for alerts.

Local testing: as with this repo's other Ceph blueprint, Ceph is a multi-daemon distributed system with no official single-container demo image (confirmed against Ceph's own containerized-deployment documentation). Use cephadm bootstrap to stand up a single-node cluster with the Dashboard enabled, then create a replicated pool with a deliberately low size to exercise this flow. The plugin's own default Dashboard port is 8443, which does not collide with Kestra's own UI on 8080.

Secrets

• CEPH_USERNAME / CEPH_PASSWORD: Dashboard account credentials used by every io.kestra.plugin.ceph.pools task in this flow.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_under_replicated_pools and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded, and read back in flows with {{ secret('NAME') }} - for example SECRET_CEPH_PASSWORD=$(echo -n '<your-local-password>' | base64). This keeps credentials out of the flow YAML but, per Kestra's own documentation, offers no encryption at rest or access control beyond the host environment; use the Enterprise secrets backend for stronger guarantees.

Inputs

• ceph_host (STRING, default localhost): Ceph Dashboard hostname or IP.
• ceph_port (INT, default 8443): Ceph Dashboard REST API port.
• min_replica_size (INT, default 3): replica-count floor for replicated pools.
• auto_remediate (SELECT: "false", "true"; default "false"): must be "true" for remediation to even be considered.
• dry_run (SELECT: "true", "false"; default "true"): must be explicitly "false", together with auto_remediate: "true", for any pool's size to actually be raised.

Outputs

• outputs.list_pools.pools / .total: the full pool list and count, every run.
• outputs.raise_pool_replica_size (inside the Loop): the updated PoolInfo for each pool actually changed, present only when remediation ran.

Quick start

1. Stand up a Ceph cluster with the Dashboard module enabled, for example via cephadm bootstrap.
2. Create a replicated pool with size deliberately below your intended minimum, for example ceph osd pool create testpool 32 32 replicated followed by ceph osd pool set testpool size 1.
3. Add the CEPH_USERNAME, CEPH_PASSWORD, and SLACK_WEBHOOK_URL secrets.
4. Run the flow manually with the defaults and confirm alert_under_replicated_pools fires, naming testpool.
5. Run once with auto_remediate: "true" and dry_run: "true" and confirm log_dry_run_remediation describes the fix without applying it.
6. Run once with auto_remediate: "true" and dry_run: "false" and confirm raise_pool_replica_size runs, then verify with ceph osd pool get testpool size that it is now 3.
7. Enable periodic_replication_check once you trust the check; it always runs in the safe auto_remediate: "false" / dry_run: "true" mode regardless of what you leave the flow's own defaults set to.

How to extend

• Add a separate check for erasure-coded pools, which this flow explicitly excludes - their durability is governed by the erasure-coding profile's k/m values, not size, and needs a different comparison entirely.
• Add a pg_num sanity check alongside size, since a pool with too few placement groups for its size can cause its own imbalance problems distinct from under-replication.
• Route pools matching a naming convention (for example a prod- prefix) to a stricter minimum than everything else, by passing a different min_replica_size per Loop iteration over a list of (pool_name_pattern, minimum) pairs.
• Chain cluster.GetHealth (from this repo's other Ceph blueprint) as a second probe, since a cluster already in HEALTH_ERR may not be the moment to also start raising replica sizes and triggering a rebalance.

Pitfalls

• This flow only evaluates replicated pools. The jq filter excludes type == "erasure" pools entirely - their durability is controlled by the erasure-coding profile, not a simple size comparison, and checking them needs different logic this flow does not implement.
• Raising size is not free. It increases storage consumption and triggers data rebalancing across OSDs - real resource and I/O cost, which is exactly why this sits behind both auto_remediate and dry_run rather than running unconditionally.
• This flow only ever raises size, never lowers it. min_replica_size is a floor, not a target - a pool already above the minimum is excluded from the jq filter entirely and is never touched by Update, even if its size is far above what you set.
• There is no structured "list under-replicated pools" task in this plugin. The filtering happens entirely in this flow's own jq expression against pools.List's full result - verified from the plugin's own source, List always returns every pool, there is no server-side threshold filter to call instead.
• Update only sends fields explicitly set. Verified from the plugin's own source: this flow sets only size, so pgNum and applicationMetadata are left exactly as they were on every pool it updates - this is deliberate, not a limitation to work around.
• auto_remediate and dry_run are independent gates, not a single boolean. Both must be "true"/"false" respectively for any pool to actually be updated; setting only one leaves the other still blocking.
• Switch case keys "true"/"false" are quoted strings. auto_remediate renders as the literal string "true" or "false"; unquoted true:/false: YAML map keys would parse as booleans instead and would not match.
• This blueprint is UNTESTED against a live Ceph cluster. pools.List/pools.Update's properties and PoolInfo's fields (pool, pool_name, type, size, pg_num, pg_placement_num, application_metadata) are taken verbatim from the plugin's source on main - but no Docker/Ceph environment or Kestra engine was run to execute this flow end to end, and the Loop-over-jq-filtered-objects pattern was not observed running live, only reasoned from this repo's own consistent fromJson(item.value) precedent across every other Loop example found in this codebase.

Links

• plugin-ceph documentation
• plugin-ceph source (pools/List.java)
• plugin-ceph source (pools/Update.java)
• Ceph Dashboard REST API documentation
• Kestra secrets (OSS vs Enterprise)