Schedule icon
Webhook icon
Loop icon
Get icon
Get icon
Script icon
Process icon
If icon
Patch icon
Set icon
Delete icon
Log icon
SlackIncomingWebhook icon

Kubernetes Off-Hours Downscaler with Restore and Savings Report

Schedule-driven Kubernetes downscaler that scales non-prod Deployments to zero overnight, saves state in KV, restores at dawn, and reports savings.

Categories
CloudInfrastructure

Non-production clusters burn money all night. This blueprint turns a weekday evening schedule into a controlled scale-down of every Deployment in the namespaces you name, saves the previous replica counts in Kestra's KV store, and restores them from the morning schedule. It is the orchestration equivalent of kube-downscaler, with the plan, the audit trail, the opt-outs and the savings report living in one execution.

How it works

  1. scale_down_evening (20:00 on weekdays) runs with mode=down; scale_up_morning (07:00) runs with mode=up. The on_demand Webhook covers "wake staging up now" moments. Flow-level concurrency: limit 1 guarantees a down and an up run never overlap.
  2. inventory (core.flow.Loop over namespaces) lists every Deployment with kubernetes.kubectl.Get and reads the namespace's saved state with core.kv.Get in the same iteration, then exposes both through the loop's outputs.
  3. plan (scripts.python.Script, Process runner) builds the change list without touching the cluster. In down mode it honours the downscaler/exclude opt-out, the downscaler/downtime-replicas per-workload floor, skips anything already at the floor, and refuses to overwrite a saved state that was never restored. In up mode it restores the saved counts, skipping Deployments that disappeared or are already back. Replica-hours removed and a blended-rate savings estimate come out as task outputs.
  4. apply (core.flow.If) runs only when dry_run is false and the plan is non-empty. scale (Loop) sends one strategic-merge kubectl.Patch per Deployment that sets spec.replicas and stamps downscaler/original-replicas and downscaler/last-action annotations, with a constant retry. remember_state then writes one KV entry per namespace (overwrite: false) after a down run, or deletes it after an up run.
  5. digest posts a single Slack message with the changes, every skipped workload and its reason, and the savings line. errors alerts if any step fails so nobody arrives to a half-scaled environment.

What you get

  • Evening scale-down and morning restore for any set of namespaces, from one flow and two schedules.
  • Exact restore: replica counts are saved per namespace in the KV store, not guessed.
  • Safety rails: dry-run by default, per-workload opt-out and floor annotations, no double-down on an unrestored namespace, no overlapping runs.
  • An in-cluster audit trail through annotations and a Slack digest with replica-hours removed and an estimated saving.
  • Flow outputs (actions, skipped, replica_hours_removed, estimated_savings_usd, applied) ready for dashboards.

Who it's for

  • Platform and SRE teams running shared dev, staging or preview clusters.
  • FinOps practitioners who need evidence of savings, not just a cron job.
  • Teams migrating off ad-hoc scripts or a kube-downscaler sidecar who want the schedule, the state and the reporting in one place.

Why orchestrate this with Kestra

A cron job that runs kubectl scale --replicas=0 forgets what the replicas were, fails silently, and cannot be paused, replayed or audited. Here the schedule, the inventory, the plan, the patches, the saved state and the notification are one execution: every step is logged, the KV store keeps the state between the evening and the morning, concurrency control prevents the two schedules racing, retries absorb API hiccups, and the same flow runs on demand with different inputs.

Prerequisites

  • A Kubernetes cluster reachable from the Kestra worker and a ServiceAccount allowed to get, list and patch Deployments in the target namespaces (and nothing else).
  • Python 3 on the worker (the default Kestra image has it) or switch the plan task to the Docker task runner.
  • A Slack incoming webhook.

Secrets

  • K8S_MASTER_URL: Kubernetes API server URL.
  • K8S_TOKEN: ServiceAccount token with get, list and patch on Deployments in the target namespaces.
  • SLACK_WEBHOOK_URL: Slack incoming webhook for the digest and failure alerts.

Quick start

  1. Create the secrets above. For the ServiceAccount, bind a Role with get, list, patch on deployments.apps in each namespace and mint a token with kubectl create token.
  2. Change the Webhook key, set namespaces to your non-production namespaces and adjust the cron timezone.
  3. Annotate anything that must stay up with downscaler/exclude: "true", or give it a floor with downscaler/downtime-replicas: "1".
  4. Execute with mode=down and dry_run=true, read the Slack digest and the actions output.
  5. Execute again with dry_run=false, confirm the Deployments scaled and the annotations appeared, then run mode=up and confirm the restore.
  6. Enable both schedules together.

How to extend

  • Add StatefulSets by duplicating the deployments task with resourceType: statefulsets and merging the two lists in plan.
  • Replace the blended-rate estimate with real allocation data from Kubecost or OpenCost before the digest.
  • Add a core.flow.Pause approval in front of apply for namespaces that are shared with external testers.
  • Point on_demand at a Slack slash command or a ChatOps bot so developers can wake their environment themselves.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.