DocsWorkflow ComponentsConcurrency limits

Concurrency limits​Concurrency limits

Table of Contents

Was this page helpful?

Control concurrent executions of a given flow.

The flow level concurrency property allows you to control the number of concurrent executions of a given flow by setting the limit key.

You can treat concurrency as a global concurrency limit for that specific flow. The concurrency limit and behavior is then applied to all executions of that flow, regardless of whether those executions have been started automatically via a trigger, webhook, an API call or manually created from the UI.

For example, if you set the concurrency limit to 2, only two executions of that flow will be allowed to run at the same time. If you try to trigger a third execution, it will be queued until one of the two running executions is completed.

yaml
id: concurrency_example
namespace: example

concurrency:
  limit: 2

tasks:
  - id: log_message
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - sleep 10

As you can see in the UI, the 3rd execution has been queued while the first 2 finish their executions.

concurrency

behavior property

It's also possible to CANCEL or FAIL an execution if the concurrency limit is reached. You can customize that behavior by setting the behavior property, which is an Enum-type property with three possible values:

  • QUEUE
  • CANCEL
  • FAIL.

When concurrency is set to 2, and using the CANCEL or FAIL behavior, the third execution's state will be immediately set to CANCELLED or FAILED status respectively without running any task.

Here is an example flow that uses the concurrency property to limit the number of concurrent executions to 2. The bash task will sleep for 10 seconds, so you can trigger multiple executions of that flow and see how the concurrency property behaves.

yaml
id: concurrency_limited_flow
namespace: example

concurrency:
  behavior: FAIL # QUEUE, CANCEL or FAIL
  limit: 2 # can be any integer >= 1

tasks:
  - id: bash
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - sleep 10

As you can see in the UI, the 3rd executions failed as the first 2 executions were still running.

concurrency_fail