Many tasks in Kestra will run in dedicated Docker containers, including among others:
- Script tasks running Python, Node.js, R, Julia, Bash, and more
- Singer tasks running data ingestion syncs
- dbt tasks running dbt jobs.
Kestra uses Docker for those tasks to ensure that they run in a consistent environment and to avoid dependency conflicts.
Those tasks require a Docker daemon running on the host. Refer to the Troubleshooting guide if you encounter any issues configuring Docker.
Defining a Docker runner
To run a task in a Docker container, set the runner
property to DOCKER
:
runner: DOCKER
Many tasks, including Python, use the DOCKER
runner by default.
Using the docker
property, you can define the Docker image for the task. You can use any image from a public or private container registry, as well as a custom local image built from a Dockerfile. You may even build a Docker image using the Docker plugin in one task, and reference the built image by the tag in a downstream task.
docker:
image: ghcr.io/kestra-io/pydata:latest
The docker
property also allows you to configure credentials
with nested username
and password
properties to authenticate with a private container registry.
id: private_docker_image
namespace: dev
tasks:
- id: custom_image
type: io.kestra.plugin.scripts.python.Script
docker:
image: ghcr.io/your_org/your_package:tag
credentials:
username: your_organization
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
script: |
print("this runs using a private container image")
The task documentation of each script task provides more details about available docker
properties.
For more examples of running scripts in Docker containers, check the Script tasks page.
Next steps
Congrats! 🎉 You've completed the tutorial.
Next, you can dive into:
- Key concepts
- Developer Guide that includes a more detailed explanation of everything covered in this tutorial
- Plugins to integrate with external systems
- Deployments to deploy Kestra to production.