PLUGINSCORETASKSFLOWSIO.KESTRA.CORE.TASKS.FLOWS.WORKINGDIRECTORY

​Working​Directory

WorkingDirectory WorkingDirectory

yaml
type: "io.kestra.core.tasks.flows.WorkingDirectory"

Run tasks sequentially in the same working directory

Tasks are stateless by default. Kestra will launch each task within a temporary working directory on a Worker. The WorkingDirectory task allows reusing the same file system's working directory across multiple tasks so that multiple sequential tasks can use output files from previous tasks without having to use the outputs.taskId.outputName syntax. Note that the WorkingDirectory only works with runnable tasks because those tasks are executed directly on the Worker. This means that using flowable tasks such as the Parallel task within the WorkingDirectory task will not work. The WorkingDirectory task requires Kestra>=0.9.0.

Examples

Clone a git repository into the Working Directory and run a Python script

yaml
id: gitPython
namespace: dev

tasks:
  - id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: cloneRepository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/examples
        branch: main
      - id: python
        type: io.kestra.plugin.scripts.python.Commands
        docker:
          image: ghcr.io/kestra-io/pydata:latest
        commands:
          - python scripts/etl_script.py

Add input and output files within a Working Directory to use them in a Python script

yaml
id: apiJSONtoMongoDB
namespace: dev

tasks:
- id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
    - id: demoSQL
        type: io.kestra.core.tasks.storages.LocalFiles
        inputs:
        query.sql: |
            SELECT sum(total) as total, avg(quantity) as avg_quantity
            FROM sales;

    - id: inlineScript
        type: io.kestra.plugin.scripts.python.Script
        runner: DOCKER
        docker:
        image: python:3.11-slim
        beforeCommands:
        - pip install requests kestra > /dev/null
        warningOnStdErr: false
        script: |
        import requests
        import json
        from kestra import Kestra

        with open('query.sql', 'r') as input_file:
            sql = input_file.read()

        response = requests.get('https://api.github.com')
        data = response.json()

        with open('output.json', 'w') as output_file:
            json.dump(data, output_file)

        Kestra.outputs({'receivedSQL': sql, 'status': response.status_code})

    - id: jsonFiles
        type: io.kestra.core.tasks.storages.LocalFiles
        outputs:
        - output.json

- id: loadToMongoDB
    type: io.kestra.plugin.mongodb.Load
    connection:
    uri: mongodb://host.docker.internal:27017/
    database: local
    collection: github
    from: "{{outputs.jsonFiles.uris['output.json']}}"
yaml
id: working-directory
namespace: io.kestra.tests

tasks:
  - id: working-directory
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: first
        type: io.kestra.plugin.scripts.shell.Commands
        commands:
        - 'echo "{{ taskrun.id }}" > {{ workingDir }}/stay.txt'
      - id: second
        type: io.kestra.plugin.scripts.shell.Commands
        commands:
        - |
          echo '::{"outputs": {"stay":"'$(cat {{ workingDir }}/stay.txt)'"}}::'

A working directory with a cache of the node_modules directory

yaml
id: node-with-cache
namespace: dev
tasks:
  - id: working-dir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    cache:
      patterns:
        - node_modules/**
      ttl: PT1H
    tasks:
    - id: script
      type: io.kestra.plugin.scripts.node.Script
      beforeCommands:
        - npm install colors
      script: |
        const colors = require("colors");
        console.log(colors.red("Hello"));

Properties

tasks

  • Type: array
  • SubType: Task
  • Dynamic: ❌
  • Required: ✔️
  • Min items: 1

cache

  • Type: Cache
  • Dynamic: ❌
  • Required: ❌

Cache configuration

When a cache is configured, an archive of the files denoted by the cache configuration is created at the end of the execution of the task and saved in Kestra's internal storage. Then at the beginning of the next execution of the task, the archive of the files is retrieved and the working directory initialized with it.

errors

  • Type: array
  • SubType: Task
  • Dynamic: ❌
  • Required: ❌

List of tasks to run if any tasks failed on this FlowableTask

Definitions

Cache

patterns

  • Type: array
  • SubType: string
  • Dynamic: ❌
  • Required: ✔️

List of file glob patterns to include in the cache.

For example 'node_modules/**' will include all files of the node_modules directory including sub-directories.

ttl

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Format: duration

Cache TTL (Time To Live), after this duration the cache will be deleted.