Schedule icon
TopicList icon
OutputValues icon
Loop icon
TopicDescribe icon
If icon
SlackIncomingWebhook icon
Pause icon
Request icon
Sleep icon
Log icon

Redpanda under-replicated partition and rebalance guard

Guard Redpanda partitions with native Kafka AdminClient checks, alert on under-replicated partitions, pause for human approval, and trigger rebalancing.

Categories
DataInfrastructure

A Redpanda cluster partition can silently lose its replication redundancy following broker restarts, hardware degradations, or transient network partitions. Because Redpanda provides native wire-level compatibility with the Apache Kafka protocol, cluster health can be audited directly using standard Kafka AdminClient APIs without installing proprietary CLI agents on workers.

This blueprint transforms partition health management into a real Day-2 operational guard: io.kestra.plugin.kafka.TopicList and io.kestra.plugin.kafka.TopicDescribe inspect every non-internal topic to compare assigned replicas against in-sync replicas (ISR) and verify partition leaders. If under-replicated partitions are discovered, Kestra alerts the on-call team via Slack and halts execution behind an io.kestra.plugin.core.flow.Pause approval gate. Once an authorized engineer approves the remediation, Kestra dispatches a rebalance request to the Redpanda Admin API (POST /v1/partitions/rebalance), waits for cluster stabilization, and re-queries topic metadata to verify convergence.

How it works

  1. Topic Discovery: list_topics (io.kestra.plugin.kafka.TopicList) queries cluster metadata via the Kafka AdminClient at bootstrap_servers.
  2. Scope Filtering: filter_topics (io.kestra.plugin.core.output.OutputValues) isolates non-internal topics (excluding _schemas, __consumer_offsets, etc.) or scopes to inputs.topic if explicitly provided.
  3. Partition Metadata Inspection: describe_topics (io.kestra.plugin.core.flow.Loop over io.kestra.plugin.kafka.TopicDescribe) inspects each topic's partition count, replica lists, ISR sets, and leader broker IDs.
  4. Replication Health Evaluation: evaluate_partition_health (io.kestra.plugin.core.output.OutputValues) uses jq to identify any partition where ISR count is less than configured replicas (len(isr) < len(replicas)) or where no leader is assigned (leader == null or leader < 0).
  5. Guard Routing: guard_decision (io.kestra.plugin.core.flow.If):
    • If all partitions are healthy, log_cluster_healthy records nominal cluster status.
    • If degraded partitions exist, alert_slack_under_replicated posts an incident card with degraded partition details and direct link to the Kestra execution approval UI.
  6. Human-in-the-Loop Sign-Off: pause_for_rebalance_approval (io.kestra.plugin.core.flow.Pause) halts execution until an authorized engineer reviews the degradation and provides approval (approved: true).
  7. Remediation Execution: When approved, trigger_redpanda_rebalance (io.kestra.plugin.core.http.Request) calls POST /v1/partitions/rebalance on the Redpanda Admin API to initiate replica movement across healthy brokers.
  8. Stabilization & Post-Verification: wait_for_rebalance_stabilization (io.kestra.plugin.core.flow.Sleep) pauses for 15 seconds, followed by recheck_topics which queries topic metadata via the Kafka AdminClient to confirm recovery, and notify_rebalance_success alerts Slack of completion.
  9. Failure Escalation: Flow-level errors capture unexpected broker timeouts or HTTP connection errors and notify on-call engineers immediately.

What you get

  • Native, agentless partition auditing leveraging Redpanda's Kafka API compatibility.
  • Noise-free detection of under-replicated and leaderless partitions across all cluster topics.
  • Safety-first human approval gate (Pause) preventing unexpected rebalances during peak traffic.
  • Automated cluster partition rebalance invocation via Redpanda Admin API upon engineer sign-off.
  • End-to-end auditability and Slack notifications tracking every operational step and approval note.

Who it's for

  • Platform Engineers and SREs maintaining production Redpanda and Kafka-compatible streaming infrastructure.
  • Data Infrastructure teams seeking automated, auditable runbooks for partition replication incidents.
  • DevOps teams replacing unmonitored cron bash scripts (rpk cluster health) with declarative workflows.

Why orchestrate this with Kestra

Ad-hoc scripts running on bastion hosts cannot securely halt for human approval, lack transparent execution timelines, and create blind spots when network partitions occur. Kestra provides native Kafka and HTTP plugins, durable workflow pausing (io.kestra.plugin.core.flow.Pause), centralized secrets management, and automated Slack escalation that turns incident response into an auditable self-healing workflow.

Prerequisites

  • A running Redpanda cluster with Kafka API accessible at bootstrap_servers (port 9092) and Admin API accessible at admin_api_url (port 9644).
  • An incoming Slack webhook URI for notifications.

Quick start

  1. Set bootstrap_servers to your Redpanda Kafka broker address (e.g., redpanda.internal:9092).
  2. Set admin_api_url to your Redpanda Admin API endpoint (e.g., http://redpanda.internal:9644).
  3. Supply your Slack incoming webhook URL in slack_webhook_uri.
  4. Run the flow manually to audit cluster partition health.
  5. If under-replicated partitions are detected, open the execution URL, review degraded partitions, and resume the flow to trigger the rebalance.
  6. Enable the periodic_health_audit schedule trigger to establish continuous automated partition guarding.

How to extend

  • Topic-Specific Criticality: Add input filters or SLA thresholds to prioritize tier-1 transactional topics over bulk log topics.
  • Intra-Broker Core Balancing: Extend remediation with POST /v1/partitions/rebalance_cores to optimize CPU core distribution on multi-core brokers.
  • PagerDuty Escalation: Integrate io.kestra.plugin.pagerduty.Incident alongside Slack to page on-call SREs if leaderless partitions are detected.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.