Run a task on an on-demand Databricks cluster
Spin up a Databricks cluster on demand, run a Spark job, then tear the cluster down automatically.
Guard Redpanda partitions with native Kafka AdminClient checks, alert on under-replicated partitions, pause for human approval, and trigger rebalancing.
A Redpanda cluster partition can silently lose its replication redundancy following broker restarts, hardware degradations, or transient network partitions. Because Redpanda provides native wire-level compatibility with the Apache Kafka protocol, cluster health can be audited directly using standard Kafka AdminClient APIs without installing proprietary CLI agents on workers.
This blueprint transforms partition health management into a real Day-2 operational guard:
io.kestra.plugin.kafka.TopicList and io.kestra.plugin.kafka.TopicDescribe inspect every non-internal topic
to compare assigned replicas against in-sync replicas (ISR) and verify partition leaders. If under-replicated
partitions are discovered, Kestra alerts the on-call team via Slack and halts execution behind an
io.kestra.plugin.core.flow.Pause approval gate. Once an authorized engineer approves the remediation,
Kestra dispatches a rebalance request to the Redpanda Admin API (POST /v1/partitions/rebalance), waits for
cluster stabilization, and re-queries topic metadata to verify convergence.
list_topics (io.kestra.plugin.kafka.TopicList) queries cluster metadata via the Kafka AdminClient at bootstrap_servers.filter_topics (io.kestra.plugin.core.output.OutputValues) isolates non-internal topics (excluding _schemas, __consumer_offsets, etc.) or scopes to inputs.topic if explicitly provided.describe_topics (io.kestra.plugin.core.flow.Loop over io.kestra.plugin.kafka.TopicDescribe) inspects each topic's partition count, replica lists, ISR sets, and leader broker IDs.evaluate_partition_health (io.kestra.plugin.core.output.OutputValues) uses jq to identify any partition where ISR count is less than configured replicas (len(isr) < len(replicas)) or where no leader is assigned (leader == null or leader < 0).guard_decision (io.kestra.plugin.core.flow.If):log_cluster_healthy records nominal cluster status.alert_slack_under_replicated posts an incident card with degraded partition details and direct link to the Kestra execution approval UI.pause_for_rebalance_approval (io.kestra.plugin.core.flow.Pause) halts execution until an authorized engineer reviews the degradation and provides approval (approved: true).trigger_redpanda_rebalance (io.kestra.plugin.core.http.Request) calls POST /v1/partitions/rebalance on the Redpanda Admin API to initiate replica movement across healthy brokers.wait_for_rebalance_stabilization (io.kestra.plugin.core.flow.Sleep) pauses for 15 seconds, followed by recheck_topics which queries topic metadata via the Kafka AdminClient to confirm recovery, and notify_rebalance_success alerts Slack of completion.errors capture unexpected broker timeouts or HTTP connection errors and notify on-call engineers immediately.Pause) preventing unexpected rebalances during peak traffic.rpk cluster health) with declarative workflows.Ad-hoc scripts running on bastion hosts cannot securely halt for human approval, lack transparent execution
timelines, and create blind spots when network partitions occur. Kestra provides native Kafka and HTTP plugins,
durable workflow pausing (io.kestra.plugin.core.flow.Pause), centralized secrets management, and automated
Slack escalation that turns incident response into an auditable self-healing workflow.
bootstrap_servers (port 9092) and Admin API accessible at admin_api_url (port 9644).bootstrap_servers to your Redpanda Kafka broker address (e.g., redpanda.internal:9092).admin_api_url to your Redpanda Admin API endpoint (e.g., http://redpanda.internal:9644).slack_webhook_uri.periodic_health_audit schedule trigger to establish continuous automated partition guarding.POST /v1/partitions/rebalance_cores to optimize CPU core distribution on multi-core brokers.io.kestra.plugin.pagerduty.Incident alongside Slack to page on-call SREs if leaderless partitions are detected.