Run Shell Scripts and Shell commands in a working directory using a Process Task Runner

Chain Bash scripts and ad hoc shell commands into a single, ordered pipeline where every step reads and writes the same files. This blueprint generates a CSV, inspects it, and filters a column, all inside one shared working directory, so you never have to copy artifacts between tasks or wire up external storage just to pass a file from one step to the next.

How it works

• working_directory (io.kestra.plugin.core.flow.WorkingDirectory) creates a single scratch directory that all child tasks share, so files written by one task are visible to the others.
• create_csv_file (io.kestra.plugin.scripts.shell.Script) runs an inline Bash script that writes a header and ten rows of random values into file.csv.
• inspect_file (io.kestra.plugin.scripts.shell.Commands) runs cat file.csv to print the generated content.
• filter_file (io.kestra.plugin.scripts.shell.Commands) runs cut -d ',' -f 2 file.csv | head -n 6 to extract the first rows of the second column.
• Every script task uses the io.kestra.plugin.core.runner.Process task runner, executing commands directly on the Kestra worker host without containers.

What you get

• A working pattern for sharing files across multiple shell steps.
• A mix of full Script blocks and one-line Commands, so you can see both styles.
• Fast local execution with no Docker or remote infrastructure required.
• A reusable skeleton for any file-producing, file-consuming shell pipeline.

Who it's for

• Data engineers gluing together CSV, log, or text-processing steps.
• Platform and DevOps engineers migrating cron-driven Bash jobs into an orchestrator.
• Anyone prototyping a shell workflow before adding heavier tooling.

Why orchestrate this with Kestra

A raw shell script or a cron line gives you no visibility, no retries, and no shared state between steps. Kestra wraps these same commands in declarative YAML with event and schedule triggers, automatic retries, per-task logs, and execution lineage so you can see exactly which step produced which file. The WorkingDirectory task solves the gap a plain shell runner cannot: it guarantees ordered tasks operate on a common filesystem while keeping each step independently observable and rerunnable.

Prerequisites

• A running Kestra instance.
• A worker host with bash, cat, cut, and head available (standard on Linux and macOS).

Secrets

• None. This blueprint references no {{ secret('NAME') }} values.

Quick start

1. Add the blueprint to a Kestra namespace.
2. Execute the flow manually from the UI.
3. Open the inspect_file task logs to view the generated CSV.
4. Open the filter_file task logs to confirm the filtered column output.

How to extend

• Replace the random-data script with a real download or extract step.
• Add tasks that read file.csv and load it into a database or warehouse.
• Swap the Process task runner for a Docker task runner to pin dependencies.
• Add a Schedule trigger to run the pipeline on a recurring basis.

Links

• https://kestra.io/plugins/plugin-script-shell