WorkingDirectory
WorkingDirectory
type: "io.kestra.core.tasks.flows.WorkingDirectory"
Run tasks sequentially in the same working directory.
Tasks are stateless by default. Kestra will launch each task within a temporary working directory on a Worker. The WorkingDirectory
task allows reusing the same file system's working directory across multiple tasks so that multiple sequential tasks can use output files from previous tasks without having to use the outputs.taskId.outputName
syntax. Note that the WorkingDirectory
only works with runnable tasks because those tasks are executed directly on the Worker. This means that using flowable tasks such as the Parallel
task within the WorkingDirectory
task will not work. The WorkingDirectory
task requires Kestra>=0.9.0.
Examples
Clone a git repository into the Working Directory and run a Python script.
id: gitPython
namespace: dev
tasks:
- id: wdir
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: cloneRepository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/examples
branch: main
- id: python
type: io.kestra.plugin.scripts.python.Commands
docker:
image: ghcr.io/kestra-io/pydata:latest
commands:
- python scripts/etl_script.py
Add input and output files within a Working Directory to use them in a Python script.
id: apiJSONtoMongoDB
namespace: dev
tasks:
- id: wdir
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: demoSQL
type: io.kestra.core.tasks.storages.LocalFiles
inputs:
query.sql: |
SELECT sum(total) as total, avg(quantity) as avg_quantity
FROM sales;
- id: inlineScript
type: io.kestra.plugin.scripts.python.Script
runner: DOCKER
docker:
image: python:3.11-slim
beforeCommands:
- pip install requests kestra > /dev/null
warningOnStdErr: false
script: |
import requests
import json
from kestra import Kestra
with open('query.sql', 'r') as input_file:
sql = input_file.read()
response = requests.get('https://api.github.com')
data = response.json()
with open('output.json', 'w') as output_file:
json.dump(data, output_file)
Kestra.outputs({'receivedSQL': sql, 'status': response.status_code})
- id: jsonFiles
type: io.kestra.core.tasks.storages.LocalFiles
outputs:
- output.json
- id: loadToMongoDB
type: io.kestra.plugin.mongodb.Load
connection:
uri: mongodb://host.docker.internal:27017/
database: local
collection: github
from: "{{ outputs.jsonFiles.uris['output.json'] }}"
id: working-directory
namespace: io.kestra.tests
tasks:
- id: working-directory
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: first
type: io.kestra.plugin.scripts.shell.Commands
commands:
- 'echo "{{ taskrun.id }}" > {{ workingDir }}/stay.txt'
- id: second
type: io.kestra.plugin.scripts.shell.Commands
commands:
- |
echo '::{"outputs": {"stay":"'$(cat {{ workingDir }}/stay.txt)'"}}::'
A working directory with a cache of the node_modules directory.
id: node-with-cache
namespace: dev
tasks:
- id: working-dir
type: io.kestra.core.tasks.flows.WorkingDirectory
cache:
patterns:
- node_modules/**
ttl: PT1H
tasks:
- id: script
type: io.kestra.plugin.scripts.node.Script
beforeCommands:
- npm install colors
script: |
const colors = require("colors");
console.log(colors.red("Hello"));
Properties
tasks
- Type: array
- SubType: Task
- Dynamic: ❌
- Required: ✔️
- Min items:
1
cache
- Type: WorkingDirectory-Cache
- Dynamic: ❌
- Required: ❌
Cache configuration.
When a cache is configured, an archive of the files denoted by the cache configuration is created at the end of the execution of the task and saved in Kestra's internal storage. Then at the beginning of the next execution of the task, the archive of the files is retrieved and the working directory initialized with it.
errors
- Type: array
- SubType: Task
- Dynamic: ❌
- Required: ❌
List of tasks to run if any tasks failed on this FlowableTask.
namespaceFiles
- Type: NamespaceFiles
- Dynamic: ❌
- Required: ❌
Inject namespace files.
Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the
include
orexclude
properties to limit which namespace files will be injected.
Definitions
io.kestra.core.models.tasks.NamespaceFiles
Properties
enabled
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
true
Whether to enable namespace files to be loaded into the working directory. If explicitly set to true
in a task, it will load all Namespace Files into the task's working directory. Note that this property is by default set to true
so that you can specify only the include
and exclude
properties to filter the files to load without having to explicitly set enabled
to true
.
exclude
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
A list of filters to exclude matching glob patterns. This allows you to exclude a subset of the Namespace Files from being downloaded at runtime. You can combine this property together with include
to only inject a subset of files that you need into the task's working directory.
include
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
A list of filters to include only matching glob patterns. This allows you to only load a subset of the Namespace Files into the working directory.
io.kestra.core.tasks.flows.WorkingDirectory-Cache
Properties
patterns
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ✔️
List of file glob patterns to include in the cache.
For example, 'node_modules/**' will include all files of the node_modules directory including sub-directories.
ttl
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
Cache TTL (Time To Live), after this duration the cache will be deleted.