PLUGINSCORETASKSSTORAGESIO.KESTRA.CORE.TASKS.STORAGES.LOCALFILES

LocalFiles

Table of Contents

Contribute

LocalFiles

yaml

type: "io.kestra.core.tasks.storages.LocalFiles"

Allow to create files in the local filesystem or to send files from the local filesystem to the internal storage.

This task should be used with the WorkingDirectory task to be able to access the same local filesystem within multiple tasks. Note that this task cannot be skipped, so setting disabled: true will not work on this task.

Examples

Output local files created in a Python task and load them to S3

yaml

id: outputsFromPythonTask
namespace: dev

tasks:
  - id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: cloneRepository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/examples
        branch: main

      - id: gitPythonScripts
        type: io.kestra.plugin.scripts.python.Commands
        warningOnStdErr: false
        runner: DOCKER
        docker:
          image: ghcr.io/kestra-io/pydata:latest
        beforeCommands:
          - pip install faker > /dev/null
        commands:
          - python scripts/etl_script.py
          - python scripts/generate_orders.py

      - id: outputFile
        type: io.kestra.core.tasks.storages.LocalFiles
        outputs:
          - orders.csv
          - "*.parquet"

  - id: loadCsvToS3
    type: io.kestra.plugin.aws.s3.Upload
    accessKeyId: "{{secret('AWS_ACCESS_KEY_ID')}}"
    secretKeyId: "{{secret('AWS_SECRET_ACCESS_KEY')}}"
    region: eu-central-1
    bucket: kestraio
    key: stage/orders.csv
    from: "{{outputs.outputFile.uris['orders.csv']}}"
    disabled: true

Create a local file that will be accessible to a bash task

yaml

id: "local-files"
namespace: "io.kestra.tests"

tasks:
  - id: workingDir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
    - id: inputFiles
      type: io.kestra.core.tasks.storages.LocalFiles
      inputs:
        hello.txt: "Hello World\n"
        address.json: "{{ outputs.myTaskId.uri }}"
    - id: bash
      type: io.kestra.plugin.scripts.shell.Commands
      commands:
        - cat hello.txt

Send local files to Kestra's internal storage

yaml

id: "local-files"
namespace: "io.kestra.tests"

tasks:
  - id: workingDir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
    - id: bash
      type: io.kestra.plugin.scripts.shell.Commands
      commands:
        - mkdir -p sub/dir
        - echo "Hello from Bash" >> sub/dir/bash1.txt
        - echo "Hello from Bash" >> sub/dir/bash2.txt
    - id: outputFiles
      type: io.kestra.core.tasks.storages.LocalFiles
      outputs:
        - sub/**