How the Kestra Executor Orchestrates Executions
For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append.mdto anykestra.io/docs/*URL for plain Markdown.
The Executor is the server component that moves every execution from start to finish.
It decides which task runs next, sends runnable tasks to Workers through the Worker Controller, runs flowable tasks itself, starts subflows, and evaluates Flow triggers. It also applies the rules a flow declares: concurrency limits, quotas, retries, pauses, SLAs, and kills. Each section below links to the page that documents how to configure the feature.
What the Executor does
Every change to an execution reaches the Executor as a message on the queue: a new execution, a task result from a Worker, a kill request, a child execution ending. For each message, the Executor loads the execution, works out its next state, and publishes what must happen next.
When a message changes the execution, the Executor saves it and sends itself an event that the execution changed. Processing that event can change the execution again, which produces another event. This loop continues until a pass leaves the execution unchanged, so one message can move an execution through several steps.
The Executor treats three kinds of tasks differently:
- Runnable tasks, such as a script or a query, are sent to a Worker. Their task run moves to
SUBMITTEDuntil a Worker picks it up. - Flowable tasks, such as
Parallel,If, orLoop, run inside the Executor. They only decide which tasks run next, so their task run moves straight toRUNNING. Subflowtasks also run inside the Executor, which creates their child execution and moves the task run toRUNNING. See Subflows.
Messages for the same execution are applied one at a time, in order, while different executions are processed in parallel. A duplicate or outdated task result is ignored, so a result delivered twice doesn’t run the next tasks twice.
Each Executor process also runs an embedded system Worker. It serves a reserved queue that system tasks always use: the PurgeExecutions, PurgeFlows, PurgeLogs, and PurgeStorage tasks, and the Enterprise Edition PurgeAuditLogs, log shipper, and AssetShipper tasks.
Only the Executor writes executions
The Executor is the only component that creates or updates executions. Other components send it a command and let it apply the change. This includes creating an execution from the API, a Schedule trigger, or a Webhook trigger, and every action on an existing execution: restart, replay, pause, resume, resume from a breakpoint, force run, unqueue, changing a task run’s state, and updating labels or status. A kill is sent as a separate kill request, which the Executor also applies.
Deleting an execution and running the PurgeExecutions task are the exceptions: they remove executions from the repository directly.
How an execution moves forward
When an execution is created, the Executor checks it before running anything:
- An execution with a future
scheduleDatestaysCREATEDuntil that date, then starts. - If the flow can’t be parsed, or a governance rule blocks it, the execution fails immediately and the reason appears in its logs.
- An execution created in a final state, for example because its trigger inputs failed to render, is saved in that state and not processed further.
As tasks complete, the Executor picks the next ones:
- A task whose
runIfevaluates to false becomesSKIPPEDand never reaches a Worker. IfrunIfcan’t be rendered, the task run fails. - When a breakpoint matches a task, that task run and the execution move to
BREAKPOINTand nothing more is dispatched until you resume. Breakpoints are what the Playground uses to run a flow one task at a time.
In Enterprise Edition, the Executor routes each runnable task to a worker group, resolving the selector from the task, then the flow, its namespaces, and the tenant. If no worker group matches, the task fails. If the worker group has no running Workers, its fallback decides whether the task waits, fails, is canceled, or runs on the default Workers.
Concurrency limits, SLAs, and quotas
Before a new or restarted execution runs, the Executor admits it in this order:
- It registers a monitor for each time-based SLA, such as a maximum duration.
- It checks quotas.
- It checks concurrency limits.
For concurrency, the Executor checks the flow’s limit first, then any namespace and tenant limits, as described in evaluation order. The first limit that is full decides the outcome with its behavior: QUEUE moves the execution to QUEUED, CANCEL cancels it, and FAIL fails it. An admitted execution holds one slot at every level that has a limit, and releases those slots when it ends.
When a slot frees up, the oldest queued execution starts first. When the freed slot belongs to a namespace or tenant limit, the next execution can come from any flow under that limit, not only the flow that just finished. Executions you start with Unqueue or Force run skip admission and don’t take a slot.
Quotas count executions in fixed time windows aligned to UTC. For example, a PT1H window starts on the hour. When an execution exceeds a quota, it fails or is canceled according to the quota’s behavior.
Quotas, and namespace and tenant concurrency limits, are available in Enterprise Edition. In the open-source edition, concurrency limits apply per flow, and a flow that declares quotas is rejected when you save it.
The Executor evaluates SLAs in two ways:
MAX_DURATIONis measured from the start of the execution by the monitor registered at admission, and checked about once per second.EXECUTION_ASSERTIONis checked each time the Executor processes an update to the running execution.
An SLA that can’t be evaluated counts as violated, and the reason is logged. When an SLA is violated, its behavior applies: FAIL or CANCEL sets the last running task run and the execution to FAILED or CANCELLED and kills the rest of the execution, including its children. NONE only adds the SLA’s labels. If several SLAs are violated in the same update, only the first changes the execution.
Retries, pauses, and waits
Anything that must happen later, such as a retry, the end of a pause, or the next LoopUntil check, is stored as a delay. The Executor checks for due delays about once per second.
For a failed task, the Executor looks for a retry policy on the task, then on its parent tasks, then on the flow:
RETRY_FAILED_TASK(the default) retries the task in the same execution. The task run showsRETRYINGwhile it waits.CREATE_NEW_EXECUTIONmarks the executionRETRIEDand starts a new one.- A task inside a
WorkingDirectoryis retried by running the wholeWorkingDirectoryagain.
Retry policies apply to runnable tasks and to Subflow. If the execution is being killed or has already ended when a retry comes due, the retry is skipped.
A Pause task with a pauseDuration moves the execution to the state set by its behavior when the duration ends. With only a timeout, the task fails when the timeout ends. A pause that was already resumed or killed isn’t resumed again by its timer.
Subflows and Flow triggers
The Executor coordinates executions that start other executions.
Subflows. A Subflow task moves to RUNNING when its child execution is created. With wait: false, it succeeds immediately and ignores the child’s outcome. Otherwise, it waits for the child to end and takes the child’s final state and outputs, adjusted by transmitFailed, allowFailure, and allowWarning. A killed child marks the Subflow task run, and the flowable tasks around it in the parent, as KILLED. A child that is retried with CREATE_NEW_EXECUTION doesn’t report back; its replacement does.
Execution depth. Each Subflow task or Flow trigger in a chain adds one level of depth, up to kestra.execution.depth.max-depth (default 100). Past the limit, a Subflow task fails with an error, and a Flow trigger is skipped with a warning in the upstream execution’s logs. The limit stops chains that would otherwise trigger each other forever.
Flow triggers. When an execution changes state, the Executor evaluates every Flow trigger against the latest revision of each flow. It skips disabled flows and triggers, draft revisions, flows that can’t be parsed, and test and playground executions. A flow never triggers itself. If a trigger’s condition can’t be rendered, the trigger doesn’t fire: a warning appears in the upstream execution’s logs and an error in the downstream flow’s logs.
A single update can move an execution through several states at once, for example from PAUSED to RUNNING to SUCCESS. A Flow trigger that listens for any of those states fires once for each.
Kills and fail-fast
When you kill an execution, it moves to KILLING:
- Task runs that haven’t started, or are waiting to retry, become
KILLEDimmediately. - A
QUEUEDexecution leaves the queue and never starts. - A
PAUSEDexecution is resumed and then killed. - The Executor tells Workers to stop the task runs they are running. The execution moves to
KILLEDonce every task run has finished.
Once the execution is killed, only its afterExecution tasks still run. Its errors and finally tasks don’t.
A kill also stops the executions that the killed execution started: subflows, and running executions started by a Flow trigger that listens to it.
Kill switch
The kill switch is available in Enterprise Edition.
A kill switch stops executions without killing them one by one. It targets a tenant, a namespace and its child namespaces, a flow, or a list of execution IDs, for a time window. The Executor checks every message it receives against the active kill switches and applies the matching one’s type:
KILLkills the execution, as described above, and labels itsystem.killSwitch: killed.CANCELsets the execution toCANCELLEDright away and labels itsystem.killSwitch: cancelled. Results from tasks that were still running are discarded.IGNOREstops processing the execution and only logs it. A new execution that matches is saved with the labelsystem.killSwitch: ignored.
A new execution that matches KILL or CANCEL is saved directly as KILLED or CANCELLED, without running.
Fail-fast
Parallel and Dag can also stop their own tasks early with onChildFailure. When a child fails with no retry left, CANCEL or FAIL interrupts every task still running under that Parallel or Dag, at any depth, and gives them that state. The Parallel or Dag itself still ends FAILED, and its errors and finally tasks still run. The accepted values are CONTINUE, CANCEL, and FAIL:
| Task | Default onChildFailure | Effect |
|---|---|---|
Parallel | FAIL | The first failure stops the other branches. |
Dag | CONTINUE | Other tasks keep running after a failure. |
When an execution ends
When an execution reaches a final state, the Executor finishes it in this order:
- Fires the Flow triggers that listen for that state.
- Releases the execution’s concurrency slots and starts the next queued execution.
- Reports the result to the parent execution, for a subflow.
- Clears the execution’s pending SLA monitors.
- Releases the trigger lock, so a Schedule trigger can start its next execution.
- Publishes the final state to the UI and to notifications.
If the flow’s outputs can’t be rendered, the execution ends FAILED, unless it was killed.
Running the Executor
Scaling
You can run several Executors side by side for availability or higher execution volume. With a JDBC backend, each execution is locked while an Executor processes it, so only one Executor works on a given execution at a time. See high availability for deployment patterns.
Each Executor starts two thread pools, one for execution messages and one for task results. Set their size with kestra.executor.thread-count. The default, 0, sizes each pool to the number of allocated CPU cores, with a minimum of 4.
Liveness
Every Executor also coordinates server liveness for the whole cluster. On each check, it marks servers that missed their heartbeats as disconnected, resubmits the tasks of Workers that stopped responding, reclaims tasks orphaned by Workers that left the cluster, and rebalances scheduling work across schedulers. How resubmission works is described in worker job resubmission.
Maintenance mode
Maintenance mode is available in Enterprise Edition.
Maintenance mode pauses the Executor:
- Requests to create or restart executions are held, and running executions don’t move to their next task.
- Retry, pause, and SLA timers wait.
- Liveness checks wait, so no tasks are resubmitted.
When you exit maintenance mode, the Executor resumes where it stopped. An Executor that starts while the instance is in maintenance mode starts paused.
Configuration
| Property | Default | Purpose |
|---|---|---|
kestra.executor.thread-count | 0 | Size of each Executor thread pool. 0 means the number of allocated CPU cores, with a minimum of 4. |
kestra.execution.depth.max-depth | 100 | Maximum depth of a chain of subflows and Flow triggers. |
kestra.task.outputs.limit | -1 | Size above which task run outputs are stored in internal storage instead of the database. -1 disables the limit. |
kestra.execution.outputs.limit | -1 | The same limit for flow outputs. |
Monitoring and troubleshooting
The Executor publishes metrics under kestra.executor.*, including executions started and ended, executions queued by concurrency limits, SLA violations, kills, and worker tasks resubmitted. See the Executor metrics for the full list.
To trace Executor activity with OpenTelemetry, use the io.kestra.executor.DefaultExecutor category. See OpenTelemetry.
When a specific execution, flow, namespace, or tenant blocks the Executor, start it with the --ignore-* flags to skip that work while you investigate. See the kill switch CLI flags.
Was this page helpful?