DocsWorkflow Components

Concurrency Limits​Concurrency ​Limits

Available on: >= 0.13.0

Control how many executions of a flow can run at the same time.

The flow-level concurrency property lets you limit how many executions of a flow can run concurrently by setting the limit key.

Think of concurrency as a global execution limit for that specific flow. The concurrency limit and behavior is then applied to all executions of that flow, regardless of whether those executions have been started automatically via a trigger, webhook, an API call, or manually created from the UI.

For example, if you set the concurrency limit to 2, only two executions of that flow will be allowed to run at the same time. If you try to trigger a third execution, it will be queued until one of the two running executions is completed.

yaml
id: concurrency_example
namespace: company.team

concurrency:
  limit: 2

tasks:
  - id: wait
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - sleep 10

In the UI, the third execution is queued while the first two finish successfully.

concurrency

behavior property

You can customize the behavior when the concurrency limit is reached by choosing to queue, cancel, or fail the new execution. To do that, set the behavior Enum-type property to one of the following values:

  • QUEUE
  • CANCEL
  • FAIL

For example, with concurrency.limit set to 2 and CANCEL or FAIL behavior, the third execution is immediately marked as CANCELLED or FAILED without running any tasks.

Below is a full flow example that uses the concurrency property to limit the number of concurrent executions to 2. The bash task sleeps for 10 seconds, so you can trigger multiple executions of that flow and see how the concurrency property behaves.

yaml
id: concurrency_limited_flow
namespace: company.team

concurrency:
  behavior: FAIL # QUEUE, CANCEL or FAIL
  limit: 2 # can be any integer >= 1

tasks:
  - id: wait
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - sleep 10

As you can see in the UI, the third execution failed as the first two executions were still running.

concurrency_fail

Tracking Concurrency Slots from the UI

The Concurrency tab on the Flow page lets you track and troubleshoot concurrency issues. Introduced in Kestra 0.19.0, it shows a progress bar with the number of active slots compared to the total slots available. Below that progress bar, you can see a table showing currently running and queued Executions, providing a clear overview of the flow's concurrency status.

concurrency_page_1

To see the concurrency behavior in action, you can configure a flow with a concurrency limit as follows:

yaml
id: concurrent
namespace: company.team

concurrency:
  behavior: QUEUE
  limit: 5

tasks:
  - id: long_running_task
    type: io.kestra.plugin.scripts.shell.Commands
    commands:
      - sleep 90
    taskRunner:
      type: io.kestra.plugin.core.runner.Process

Next, trigger multiple Executions of that flow and watch the Concurrency tab showing the active slots and queued Executions.

concurrency_page_2

How to troubleshoot Concurrency issues

Imagine that you encounter a situation where the concurrency limit is reached, and some executions are stuck in the QUEUED state. Here are some steps to troubleshoot and resolve the issue.

Check the Concurrency tab

The Concurrency tab on the Flow UI page described above allows you to see which executions are RUNNING and which are QUEUED (i.e., waiting or stuck). This page can help you troubleshoot which Executions are taking concurrency slots and which are waiting to be processed.

In the future, this page will also let you run stuck executions while ignoring concurrency limits.

Edit the Concurrency property

You can edit the concurrency property within the flow (or remove that property entirely to get rid of any limits) and Save the flow code. The modified concurrency limit and behavior will be immediately taken into account for all Executions in progress because the Executor checks this for the latest flow revision rather than for the revision of the Execution.

Was this page helpful?