Run multiple subflows in parallel and wait for their completion - use taskDefaults to avoid boilerplate code

Source

yaml

id: parallel-subflows
namespace: company.team

tasks:
  - id: parallel
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: flow1
        type: io.kestra.plugin.core.flow.Subflow
        flowId: flow1
        namespace: company.team

      - id: flow2
        type: io.kestra.plugin.core.flow.Subflow
        flowId: flow2
        namespace: company.team

      - id: flow3
        type: io.kestra.plugin.core.flow.Subflow
        flowId: flow3
        namespace: company.team

pluginDefaults:
  - type: io.kestra.plugin.core.flow.Subflow
    values:
      namespace: company.team
      wait: true
      transmitFailed: true

About this blueprint

Core

Add the child flows first:

First flow:

yaml

id: flow1
namespace: company.team

tasks:
  - id: get
    type: io.kestra.plugin.core.debug.Return
    format: hi from {{ flow.id }}

Second flow:

yaml

id: flow2
namespace: company.team

tasks:
  - id: get
    type: io.kestra.plugin.core.debug.Return
    format: hi from {{ flow.id }}

Third flow:

yaml

id: flow3
namespace: company.team

tasks:
  - id: get
    type: io.kestra.plugin.core.debug.Return
    format: hi from {{ flow.id }}

Then run the parent flow parallelSubflows to trigger multiple subflows in parallel.

Parallel

Subflow

More Related Blueprints

CoreInfrastructure

Extract logs from Kestra and load them to Datadog

Getting StartedCoreData

Getting started with Kestra — a Data Engineering Pipeline example

CoreData

Fetch data from Apache Druid and transform it in Python with Pandas

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra