How the Kestra Worker Runs Tasks and Triggers

For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append .md to any kestra.io/docs/* URL for plain Markdown.

The Worker is the server component that runs your tasks and triggers.

It receives jobs from the Worker Controller, runs them, and sends the results, logs, and metrics back. Workers run your code, so they are the component you scale to add execution capacity. Each section below links to the page that documents how to configure the feature.

What the Worker runs

A Worker runs three kinds of jobs:

A realtime trigger keeps one Worker thread for as long as it runs. A Worker with eight threads and three realtime triggers has five threads left for tasks and polling triggers. Size the thread count for both, as described in Scaling.

The Worker decides nothing about what runs next. The Executor dispatches runnable tasks and the Scheduler dispatches trigger evaluations, both through the Worker Controller. A Worker never connects to the queue or the database for its work.

Kestra has two kinds of Workers:

  • Dedicated Workers, started with kestra server worker (or as part of kestra server standalone). They run all user tasks and triggers.
  • The system Worker, embedded in every Executor and standalone process. It serves a reserved queue that system tasks, such as the purge tasks, always use.

How a Worker gets work

A Worker opens one long-lived gRPC connection to a Worker Controller and tells it how much work it can accept. The controller sends jobs only up to that capacity. If the connection drops, the Worker reconnects automatically, waiting between 500 ms and 30 seconds between attempts.

A Worker’s capacity has two parts:

  • Threads. Each thread runs one job at a time. The default is eight times the number of available processors, set with -t, --thread.
  • Job buffer. Jobs the Worker has already received and holds while all its threads are busy. It defaults to the thread count, which is also its maximum. Set it with kestra.worker.job-buffer-size.

A smaller job buffer leaves more of the backlog with the Worker Controller, where a newly started Worker can pick it up instead of the work waiting on a saturated one. A buffer of 0 means the Worker only asks for a job when a thread is idle. A smaller buffer also lowers dispatch throughput for short tasks.

Running tasks

Timeouts

A task’s timeout property bounds how long it can run. When the timeout ends, the Worker stops the task and the attempt fails.

Polling-trigger evaluations have their own bound, kestra.worker.polling-trigger-timeout (default 10m). When an evaluation exceeds it, the Worker reports an error so the trigger is released and can be evaluated again. The plugin’s code keeps its thread until it returns, so the Worker runs one thread short until then. The worker.abandoned.count metric shows how many threads are held this way. Realtime triggers are exempt, since they run for as long as the trigger is active.

Kills

When an execution is killed, the Worker stops the matching tasks it is running. It remembers killed executions for 24 hours, so a task from a killed execution that arrives later is marked KILLED instead of running. Updating, disabling, or deleting a flow also stops its running realtime triggers.

Working directory and cache

Each task runs in its own temporary working directory, which the Worker deletes when the task ends. Tasks inside a WorkingDirectory task share one directory instead. The location of these directories is set by the task temporary directory configuration.

When a task uses taskCache, the Worker checks for cached outputs before running it and skips the run on a hit.

Results, logs, and metrics

The Worker sends task results, logs, and metrics to the Worker Controller as they are produced. Results are kept and resent if the controller is temporarily unreachable. Logs and metrics are best-effort: if the controller can’t receive them, they can be dropped instead of slowing the Worker down.

A task with logToFile: true writes its logs to a file that is uploaded to internal storage when the task ends, instead of streaming them.

Data access

By default, a Worker reads KV entries and namespace files directly from internal storage. To keep storage credentials off Workers, set kestra.kv.worker-access and kestra.namespace-files.worker-access to CONTROLLER on dedicated Workers. Workers then read through the Worker Controller. The default for both is STORAGE.

Task runners

A task runner changes where a script task’s code runs, for example in a Docker container or a cloud batch service, but the Worker still runs the task itself: it starts the runner, follows it, and reports the result. See task runners vs. worker groups to choose between the two.

Liveness and shutdown

A Worker sends its heartbeats to the Worker Controller. The Executor uses them to detect a Worker that stopped responding, as described in server liveness.

When a Worker stops gracefully, it:

  1. Stops accepting new jobs.
  2. Waits up to kestra.server.termination-grace-period (default 5m) for running jobs to finish.
  3. Stops any jobs still running.
  4. Sends its remaining results to the Worker Controller, for up to 30 seconds.

If any result couldn’t be delivered, the Worker reports TERMINATED_FORCED instead of TERMINATED_GRACEFULLY.

Tasks stopped at shutdown don’t report a result, so the Executor resubmits them to another Worker, according to the Worker’s kestra.server.worker-task-restart-strategy. With NEVER, those tasks aren’t resubmitted. See liveness and worker task restarts for the options.

Maintenance mode

In maintenance mode, a Worker stops accepting new jobs, lets running tasks finish, and stops its running realtime triggers so they can be started again after maintenance. A Worker that starts while the instance is in maintenance mode starts paused.

Running the Worker

Scaling

Add Workers to increase how many tasks run at once. Workers connect to the Worker Controller over gRPC (port 50051 by default), so they can run anywhere that can reach it, including other networks and regions. See performance tuning for sizing guidance.

The default thread count depends on the processors available to the JVM. kestra.allocated-cpu-cores doesn’t change it.

Each running realtime trigger holds a thread and a slot of the Worker’s capacity until it stops. Set --thread to the number of realtime triggers you expect on the Worker plus the number of tasks you want to run at once. The worker.trigger.running.count metric includes running realtime triggers.

Configuration

PropertyDefaultPurpose
-t, --thread (CLI)8 × available processorsThreads, each running one job at a time. For kestra server standalone, use --worker-thread; 0 starts no Worker.
kestra.worker.job-buffer-sizeThread countJobs held while all threads are busy.
kestra.worker.polling-trigger-timeout10mMaximum time for one polling-trigger evaluation.
kestra.kv.worker-accessSTORAGERead KV entries from storage or through the Worker Controller (CONTROLLER).
kestra.namespace-files.worker-accessSTORAGERead namespace files from storage or through the Worker Controller (CONTROLLER).
kestra.server.termination-grace-period5mHow long a stopping Worker waits for running jobs.
kestra.server.worker-task-restart-strategyAFTER_TERMINATION_GRACE_PERIODWhat happens to a stopped Worker’s tasks.

These properties tune the Worker’s gRPC connection to the Worker Controller. For the message size limit and shared Basic credentials, see the Worker Controller configuration.

PropertyDefaultPurpose
kestra.grpc.channel.keep-alive-time1hInterval between keep-alive pings, sent even when no call is active.
kestra.grpc.channel.shutdown-timeout30sMaximum wait for the connection to close gracefully.
kestra.grpc.channel.retry.enabledtrueRetry calls that are safe to repeat, such as KV and namespace file reads, when the Worker Controller is unavailable.
kestra.grpc.channel.retry.max-attempts4Attempts per retried call, including the first. A value below 2 disables retries.
kestra.grpc.channel.retry.initial-backoffPT0.5SDelay before the first retry.
kestra.grpc.channel.retry.max-backoffPT5SMaximum delay between retries.
kestra.grpc.channel.retry.backoff-multiplier2.0Factor applied to the delay after each failed attempt.
kestra.worker.controllers.health-check.enabledtrueUse the gRPC health service to stop sending calls to a Worker Controller that isn’t serving.
kestra.worker.controllers.wait-for-ready.enabledtrueWait for a Worker Controller to be ready instead of failing the call immediately.
kestra.worker.controllers.wait-for-ready.deadlinePT30SMaximum wait for a ready Worker Controller when wait-for-ready is enabled.

Monitoring and troubleshooting

Workers publish metrics under kestra.worker.*, tagged with worker_group:

  • Capacity: worker.job.thread, worker.max.concurrency (threads plus job buffer), worker.running.count, and worker.pending.count. When worker.running.count stays equal to worker.job.thread, the Worker is saturated.
  • Stuck work: worker.abandoned.count and worker.timeout.count.
  • Triggers: worker.trigger.* counters and timers.

See the Worker metrics for the full list.

Each Worker also serves a /worker endpoint on the management port (8081) that lists the jobs it is running. The endpoint is sensitive, so it requires authentication.

Was this page helpful?