​Concurrency limits

Control concurrent executions of a given flow.

The flow level concurrency property allows you to control the number of concurrent executions of a given flow by setting the limit key.

You can treat concurrency as a global concurrency limit for that specific flow. The concurrency limit and behavior is then applied to all executions of that flow, regardless of whether those executions have been started automatically via a trigger, webhook, an API call or manually created from the UI.

For example, if you set the concurrency limit to 2, only two executions of that flow will be allowed to run at the same time. If you try to trigger a third execution, it will be queued until one of the two running executions is completed.

id: concurrency_example
namespace: company.team

concurrency:
  limit: 2

tasks:
  - id: wait
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - sleep 10

As you can see in the UI, the third execution has been queued while the first two have finished successfully.

behavior property

You can customize the execution behavior to CANCEL or FAIL an execution if the concurrency limit is reached. To do that, set the behavior Enum-type property to one of the following values:

• QUEUE
• CANCEL
• FAIL.

Let's say you set the concurrency.limit to 2, and you use the CANCEL or FAIL behavior. The third execution's state will be immediately set to CANCELLED or FAILED status respectively without running any task.

Here is a full flow example that uses the concurrency property to limit the number of concurrent executions to 2. The bash task will sleep for 10 seconds, so you can trigger multiple executions of that flow and see how the concurrency property behaves.

id: concurrency_limited_flow
namespace: company.team

concurrency:
  behavior: FAIL # QUEUE, CANCEL or FAIL
  limit: 2 # can be any integer >= 1

tasks:
  - id: wait
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - sleep 10

As you can see in the UI, the third execution failed as the first two executions were still running.