Refresh Multiple Power BI Datasets with Bounded Concurrency

One dataset refresh is a task, ten are an operational headache: scripts with hardcoded id lists, refreshes that pile onto the same capacity, and no idea which one failed. This blueprint declares the whole fleet as data. A Loop iterates over workspace and dataset pairs, io.kestra.plugin.powerbi.RefreshGroupDataset refreshes each with a verified wait, and concurrencyLimit: 2 keeps simultaneous refreshes from saturating a shared capacity.

How it works

1. loop_datasets (io.kestra.plugin.core.flow.Loop) iterates over a YAML list of JSON strings, each carrying a group (workspace id) and a dataset id. Adding a dataset to the fleet is a one line change.
2. Inside the loop, refresh_dataset reads the current pair with {{ fromJson(item.value).group }} and {{ fromJson(item.value).dataset }}, authenticates with the Azure AD service principal, and refreshes the dataset. With wait: true it polls every 15 seconds for up to 45 minutes and fails on a failed refresh state.
3. concurrencyLimit: 2 runs at most two refreshes at a time, protecting Premium or Fabric capacity from a refresh stampede.
4. notify posts one Discord summary after the whole list completes; the errors block posts a distinct alert when any pair fails, and the execution view shows exactly which iteration broke.

What you get

• A dataset fleet managed as a list in version control instead of ids scattered across scripts.
• Bounded parallelism: faster than sequential, safer than all-at-once on shared capacity.
• Per-dataset verification, because every refresh waits for a terminal state instead of assuming success.
• One summary message instead of ten, with per-iteration detail preserved in the execution view.

Who it's for

• BI platform teams responsible for many semantic models across several workspaces.
• Data engineers consolidating per-dataset refresh scripts into one declarative flow.
• Capacity admins who need refresh parallelism capped, not unbounded.

Why orchestrate this with Kestra

Refreshing many datasets from Power BI's own scheduler means many independent timers with no shared view and no coordination. Kestra turns the fleet into one execution: a declarative list, controlled concurrency, per-iteration logs and states, a single summary, and a single failure alert that points at the exact broken pair.

Prerequisites

• An Azure AD app registration with Power BI API permissions granted, authenticating through the OAuth 2.0 client credentials flow.
• Service principal access enabled for Power BI in the admin portal, and the service principal added to every workspace in the list.
• Workspace and dataset ids for each model to refresh.
• A Discord incoming webhook for the summary and alerts.

Secrets

• AZURE_TENANT_ID: Azure AD tenant id.
• AZURE_CLIENT_ID: Azure AD application (client) id.
• AZURE_CLIENT_SECRET: Azure AD client secret.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the four secrets to your Kestra namespace.
2. Replace the values list with your own workspace and dataset pairs.
3. Execute the flow and watch the iterations run two at a time in the Gantt view.
4. Check the Discord summary, then add a Schedule trigger if you want the fleet refreshed on a timer.

How to extend

• Raise or lower concurrencyLimit to match your capacity headroom.
• Move the values list to a flow input so different environments refresh different fleets.
• Add a name field to each JSON object and interpolate it into task logs for friendlier iteration labels.
• Chain the fleet refresh after your warehouse load, as shown in the refresh after ELT blueprint.

Links

• Power BI plugin documentation
• Loop task documentation