About this blueprint
Local files Outputs Parallel
This example demonstrates how to process files in parallel.
In the workingDir task, we generate multiple files, and store them in the internal storage.
Next, in the each task, we process these files in parallel, where the tasks path and contents run for each of the 4 output files, resulting in 8 parallel task runs.
Instead of the bash script, you may have a Python/R/Node.js script that generates such files.
yaml
id: parallelFiles
namespace: blueprint
tasks:
- id: workingDir
type: io.kestra.core.tasks.flows.WorkingDirectory
tasks:
- id: bash
type: io.kestra.plugin.scripts.shell.Commands
runner: PROCESS
commands:
- mkdir -p out
- echo "Hello from 1" >> out/output1.txt
- echo "Hello from 2" >> out/output2.txt
- echo "Hello from 3" >> out/output3.txt
- echo "Hello from 4" >> out/output4.txt
- id: out
type: io.kestra.core.tasks.storages.LocalFiles
outputs:
- out/**
- id: each
type: io.kestra.core.tasks.flows.EachParallel
value: "{{ outputs.out.uris | jq('.[]') }}"
tasks:
- id: path
type: io.kestra.core.tasks.debugs.Return
format: "{{ taskrun.value }}"
- id: contents
type: io.kestra.plugin.scripts.shell.Commands
runner: PROCESS
commands:
- cat "{{ taskrun.value }}"