Core Plugins and tasks Split
CertifiedSplit a file from Kestra internal storage.
Core Plugins and tasks Split
Split a file from Kestra internal storage.
Splits an input file by size (bytes), line count (rows), partitions, or regex grouping (first capture group), emitting new files in internal storage. Optional separator inserted between grouped lines.
Provide exactly one split strategy at a time.
type: io.kestra.plugin.core.storage.SplitExamples
Split a file by size.
id: split_bytes
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ outputs.download.uri }}"
bytes: 5KB
Split a file by rows count.
id: split_rows
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ outputs.download.uri }}"
rows: 10
Split a file in a defined number of partitions.
id: split_partitions
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ outputs.download.uri }}"
partitions: 4
Split a file by regex pattern - group lines by log level.
id: storage_tasks
namespace: company.team
tasks:
- id: generate_logs
type: io.kestra.plugin.scripts.shell.Commands
commands:
- echo "INFO - wow
ERROR - no
INFO - ok" > logs.txt
outputFiles:
- logs.txt
- id: split
type: io.kestra.plugin.core.storage.Split
from: "{{ outputs.echo.outputFiles['logs.txt'] }}"
regexPattern: "^(\w+)"
Properties
from *string
The file to be split
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150bytes string
Split a large file into multiple chunks with a maximum file size of bytes.
Can be provided as a string in the format "10MB" or "200KB". Must be KB or higher. This allows you to process large files, slit them into smaller chunks by lines and process them in parallel. For example, MySQL by default limits the size of a query size to 16MB per query. Trying to use a bulk insert query with input data larger than 16MB will fail. Splitting the input data into smaller chunks is a common strategy to circumvent this limitation. By dividing a large data set into chunks smaller than the max_allowed_packet size (e.g., 10MB), you can insert the data in multiple smaller queries. This approach not only helps to avoid hitting the query size limit but can also be more efficient and manageable in terms of memory utilization, especially for very large datasets. In short, by splitting the file by bytes, you can bulk-insert smaller chunks of e.g. 10MB in parallel to avoid this limitation.
partitions integerstring
Split a file into a fixed number of partitioned files. For example, if you have a file with 1000 lines and you set partitions to 10, the file will be split into 10 files with 100 lines each.
regexPattern string
Split file by regex pattern. Lines are grouped by the first capture group value.
A regular expression pattern with a capture group. Lines matching this pattern will be grouped by the captured value. For example, ^(\w+) will group lines by the first word extracted from the file.
rows integerstring
A number of rows per batch. The file will then be split into chunks with that maximum number of rows.
separator string
\nThe separator used to split a file into chunks. By default, it's a newline \n character. If you are on Windows, you might want to use \r\n instead.
Outputs
uris array
The URIs of split files in Kestra's internal storage
