Azure Batch Cost Guardrail with Storage Lifecycle Cleanup

An idle Azure Batch pool burns compute cost every hour nobody notices it, and a scratch storage container with no lifecycle policy grows forever. This blueprint checks both on the same hourly run: it pulls the idle node count for a Batch pool from Azure Monitor, resizes the pool down automatically when idle nodes have stayed high for the whole window, and deletes scratch blobs past their retention prefix in the same execution. Nothing here waits on a human to click resize in the portal or run a manual storage sweep.

How it works

1. The hourly io.kestra.plugin.core.trigger.Schedule fires every hour.
2. query_pool_metric (io.kestra.plugin.azure.monitoring.Query) reads the PoolIdleNodeCount metric for the Batch account from Azure Monitor's regional metrics endpoint, averaged over the window.
3. scale_decision (io.kestra.plugin.core.flow.If) checks whether the average idle count exceeds the threshold. This is an automated gate, not a human approval step: routine pool sizing does not need a person in the loop, only genuinely destructive operations do.
4. When idle nodes are high, resize_pool (io.kestra.plugin.azure.batch.pool.Resize) scales targetDedicatedNodes and targetLowPriorityNodes down independently to the configured floor.
5. purge_cold_blobs (io.kestra.plugin.azure.storage.blob.DeleteList) runs in the same execution, deleting every blob under the expired/ prefix in the scratch container, regardless of whether the pool was resized this run.
6. The errors block alerts Slack if any step fails, so a broken metric query or a permissions issue on the storage account does not fail silently.

What you get

• Automatic pool right-sizing, no dashboard click, no forgotten manual resize.
• Cost and storage control on the same schedule and execution ID, easy to audit together.
• targetDedicatedNodes and targetLowPriorityNodes tuned independently, so spot capacity scales down first.
• A single Slack alert path if the whole guardrail run fails.

Who it's for

• Platform teams responsible for Azure Batch compute spend.
• FinOps and cost-optimization teams who want automated guardrails instead of a monthly manual review.
• Teams already using Azure Batch for HPC or render workloads who need the pool to shrink when work is light.

Why orchestrate this with Kestra

Azure Batch has autoscale formulas for compute, but nothing native ties pool sizing to a storage retention sweep in one auditable run, and a custom autoscale formula still needs somewhere to alert when it misbehaves. Kestra runs the metric check, the resize decision, and the storage cleanup as one execution with full history, retries on transient Azure API failures, and a dedicated failure path to Slack, all declared in version-controlled YAML instead of a formula string embedded in the pool configuration.

Prerequisites

• An Azure Batch account and pool to monitor and resize.
• A Storage account with a scratch container that blobs land in with an expired/ prefix once past retention.
• A service principal with Azure Monitor read access and Batch account access.

Secrets

• AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET: service principal credentials for Azure Monitor.
• AZURE_BATCH_ACCOUNT_RESOURCE_ID: full Azure resource ID of the Batch account.
• AZURE_BATCH_ACCOUNT, AZURE_BATCH_ACCESS_KEY, AZURE_BATCH_ENDPOINT: Batch account credentials for the resize call.
• AZURE_STORAGE_CONNECTION, AZURE_STORAGE_ENDPOINT: Storage account credentials for the blob purge.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the secrets above to your Kestra instance.
2. Set resourceIds, poolId, and container to your real Batch account and storage container.
3. Tune the idle-node threshold in scale_decision and the target node counts in resize_pool to your workload.
4. Enable the hourly schedule trigger.

How to extend

• Add an upper-bound check alongside the idle-node floor to scale the pool back up automatically when demand returns.
• Chain a daily rollup task that writes the resize history to a reporting table for a cost trend chart.
• Replace the fixed expired/ prefix with a computed date-based prefix if retention is date-partitioned.
• Swap the Slack failure alert for Microsoft Teams if that is where your platform team watches alerts.

Links

• Plugin docs: https://kestra.io/plugins/plugin-azure
• Azure orchestration with Kestra: https://kestra.io/orchestration/azure