Run a subflow for each value in parallel and wait for their completion — recommended pattern to iterate over hundreds or thousands of list items
About this blueprint
Parallel
First, create the following flow that we'll use as a parametrized subflow:
id: mysubflow
namespace: blueprint
inputs:
- name: myinput
type: STRING
defaults: kestra
tasks:
- id: super_task
type: io.kestra.core.tasks.debugs.Return
format: hi from {{flow.id}} using input {{ inputs.myinput }}
Then, create the flow subflowForEachValue.
This flow will trigger multiple executions of the flow mysubflow. Each execution will be triggered in parallel using input from the list of values. In this example, you should see three executions of the subflow, one with the input user1, another with the input user2 and yet another execution with the input user3.
This pattern is particularly useful if the list of values you iterate over is large. As explained in the Flow best practices documentation, it's better to execute thousands of flows, each running one task, than to trigger one flow with thousands of parallel tasks. We recommend this pattern with subflows as it's easier to scale and it makes your workflows more modular and easier to maintain.
id: subflowForEachValue
namespace: blueprint
tasks:
- id: parallel
type: io.kestra.core.tasks.flows.EachParallel
value: ["user1", "user2", "user3"]
tasks:
- id: subflow
type: io.kestra.core.tasks.flows.Flow
flowId: mysubflow
namespace: blueprint
wait: true
transmitFailed: true
inputs:
myinput: "{{ taskrun.value }}"
More Related Blueprints