How the Kestra Worker Runs Tasks and Triggers
For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append.mdto anykestra.io/docs/*URL for plain Markdown.
The Worker is the server component that runs your tasks and triggers.
It receives jobs from the Worker Controller, runs them, and sends the results, logs, and metrics back. Workers run your code, so they are the component you scale to add execution capacity. Each section below links to the page that documents how to configure the feature.
What the Worker runs
A Worker runs three kinds of jobs:
- Runnable tasks, such as scripts, queries, and API calls. Flowable tasks never reach a Worker; the Executor runs them.
- Polling triggers, which check an external system on an interval.
- Realtime triggers, which hold a connection open and react to events as they arrive.
A realtime trigger keeps one Worker thread for as long as it runs. A Worker with eight threads and three realtime triggers has five threads left for tasks and polling triggers. Size the thread count for both, as described in Scaling.
The Worker decides nothing about what runs next. The Executor dispatches runnable tasks and the Scheduler dispatches trigger evaluations, both through the Worker Controller. A Worker never connects to the queue or the database for its work.
Kestra has two kinds of Workers:
- Dedicated Workers, started with
kestra server worker(or as part ofkestra server standalone). They run all user tasks and triggers. - The system Worker, embedded in every Executor and standalone process. It serves a reserved queue that system tasks, such as the purge tasks, always use.
How a Worker gets work
A Worker opens one long-lived gRPC connection to a Worker Controller and tells it how much work it can accept. The controller sends jobs only up to that capacity. If the connection drops, the Worker reconnects automatically, waiting between 500 ms and 30 seconds between attempts.
A Worker’s capacity has two parts:
- Threads. Each thread runs one job at a time. The default is eight times the number of available processors, set with
-t, --thread. - Job buffer. Jobs the Worker has already received and holds while all its threads are busy. It defaults to the thread count, which is also its maximum. Set it with
kestra.worker.job-buffer-size.
A smaller job buffer leaves more of the backlog with the Worker Controller, where a newly started Worker can pick it up instead of the work waiting on a saturated one. A buffer of 0 means the Worker only asks for a job when a thread is idle. A smaller buffer also lowers dispatch throughput for short tasks.
In Enterprise Edition, a Worker joins a Worker Group through its registration token, and the Worker Controller sends it only jobs routed to that group’s queues.
Running tasks
Timeouts
A task’s timeout property bounds how long it can run. When the timeout ends, the Worker stops the task and the attempt fails.
Polling-trigger evaluations have their own bound, kestra.worker.polling-trigger-timeout (default 10m). When an evaluation exceeds it, the Worker reports an error so the trigger is released and can be evaluated again. The plugin’s code keeps its thread until it returns, so the Worker runs one thread short until then. The worker.abandoned.count metric shows how many threads are held this way. Realtime triggers are exempt, since they run for as long as the trigger is active.
Kills
When an execution is killed, the Worker stops the matching tasks it is running. It remembers killed executions for 24 hours, so a task from a killed execution that arrives later is marked KILLED instead of running. Updating, disabling, or deleting a flow also stops its running realtime triggers.
Working directory and cache
Each task runs in its own temporary working directory, which the Worker deletes when the task ends. Tasks inside a WorkingDirectory task share one directory instead. The location of these directories is set by the task temporary directory configuration.
When a task uses taskCache, the Worker checks for cached outputs before running it and skips the run on a hit.
Results, logs, and metrics
The Worker sends task results, logs, and metrics to the Worker Controller as they are produced. Results are kept and resent if the controller is temporarily unreachable. Logs and metrics are best-effort: if the controller can’t receive them, they can be dropped instead of slowing the Worker down.
A task with logToFile: true writes its logs to a file that is uploaded to internal storage when the task ends, instead of streaming them.
Data access
By default, a Worker reads KV entries and namespace files directly from internal storage. To keep storage credentials off Workers, set kestra.kv.worker-access and kestra.namespace-files.worker-access to CONTROLLER on dedicated Workers. Workers then read through the Worker Controller. The default for both is STORAGE.
In Enterprise Edition, kestra.secret.worker-access controls how Workers reach the secrets manager: DIRECT (the default) or through the Worker Controller with CONTROLLER or CONTROLLER_ONLY.
Task runners
A task runner changes where a script task’s code runs, for example in a Docker container or a cloud batch service, but the Worker still runs the task itself: it starts the runner, follows it, and reports the result. See task runners vs. worker groups to choose between the two.
Liveness and shutdown
A Worker sends its heartbeats to the Worker Controller. The Executor uses them to detect a Worker that stopped responding, as described in server liveness.
When a Worker stops gracefully, it:
- Stops accepting new jobs.
- Waits up to
kestra.server.termination-grace-period(default5m) for running jobs to finish. - Stops any jobs still running.
- Sends its remaining results to the Worker Controller, for up to 30 seconds.
If any result couldn’t be delivered, the Worker reports TERMINATED_FORCED instead of TERMINATED_GRACEFULLY.
Tasks stopped at shutdown don’t report a result, so the Executor resubmits them to another Worker, according to the Worker’s kestra.server.worker-task-restart-strategy. With NEVER, those tasks aren’t resubmitted. See liveness and worker task restarts for the options.
Maintenance mode
Maintenance mode is available in Enterprise Edition.
In maintenance mode, a Worker stops accepting new jobs, lets running tasks finish, and stops its running realtime triggers so they can be started again after maintenance. A Worker that starts while the instance is in maintenance mode starts paused.
Running the Worker
Scaling
Add Workers to increase how many tasks run at once. Workers connect to the Worker Controller over gRPC (port 50051 by default), so they can run anywhere that can reach it, including other networks and regions. See performance tuning for sizing guidance.
The default thread count depends on the processors available to the JVM. kestra.allocated-cpu-cores doesn’t change it.
Each running realtime trigger holds a thread and a slot of the Worker’s capacity until it stops. Set --thread to the number of realtime triggers you expect on the Worker plus the number of tasks you want to run at once. The worker.trigger.running.count metric includes running realtime triggers.
Configuration
| Property | Default | Purpose |
|---|---|---|
-t, --thread (CLI) | 8 × available processors | Threads, each running one job at a time. For kestra server standalone, use --worker-thread; 0 starts no Worker. |
kestra.worker.job-buffer-size | Thread count | Jobs held while all threads are busy. |
kestra.worker.polling-trigger-timeout | 10m | Maximum time for one polling-trigger evaluation. |
kestra.kv.worker-access | STORAGE | Read KV entries from storage or through the Worker Controller (CONTROLLER). |
kestra.namespace-files.worker-access | STORAGE | Read namespace files from storage or through the Worker Controller (CONTROLLER). |
kestra.server.termination-grace-period | 5m | How long a stopping Worker waits for running jobs. |
kestra.server.worker-task-restart-strategy | AFTER_TERMINATION_GRACE_PERIOD | What happens to a stopped Worker’s tasks. |
These properties tune the Worker’s gRPC connection to the Worker Controller. For the message size limit and shared Basic credentials, see the Worker Controller configuration.
| Property | Default | Purpose |
|---|---|---|
kestra.grpc.channel.keep-alive-time | 1h | Interval between keep-alive pings, sent even when no call is active. |
kestra.grpc.channel.shutdown-timeout | 30s | Maximum wait for the connection to close gracefully. |
kestra.grpc.channel.retry.enabled | true | Retry calls that are safe to repeat, such as KV and namespace file reads, when the Worker Controller is unavailable. |
kestra.grpc.channel.retry.max-attempts | 4 | Attempts per retried call, including the first. A value below 2 disables retries. |
kestra.grpc.channel.retry.initial-backoff | PT0.5S | Delay before the first retry. |
kestra.grpc.channel.retry.max-backoff | PT5S | Maximum delay between retries. |
kestra.grpc.channel.retry.backoff-multiplier | 2.0 | Factor applied to the delay after each failed attempt. |
kestra.worker.controllers.health-check.enabled | true | Use the gRPC health service to stop sending calls to a Worker Controller that isn’t serving. |
kestra.worker.controllers.wait-for-ready.enabled | true | Wait for a Worker Controller to be ready instead of failing the call immediately. |
kestra.worker.controllers.wait-for-ready.deadline | PT30S | Maximum wait for a ready Worker Controller when wait-for-ready is enabled. |
Monitoring and troubleshooting
Workers publish metrics under kestra.worker.*, tagged with worker_group:
- Capacity:
worker.job.thread,worker.max.concurrency(threads plus job buffer),worker.running.count, andworker.pending.count. Whenworker.running.countstays equal toworker.job.thread, the Worker is saturated. - Stuck work:
worker.abandoned.countandworker.timeout.count. - Triggers:
worker.trigger.*counters and timers.
See the Worker metrics for the full list.
Each Worker also serves a /worker endpoint on the management port (8081) that lists the jobs it is running. The endpoint is sensitive, so it requires authentication.
Was this page helpful?