Core Plugins and tasks Split

Core Plugins and tasks Split
Certified
Split a file from Kestra internal storage.

Splits an input file by size (bytes), line count (rows), partitions, or regex grouping (first capture group), emitting new files in internal storage. Optional separator inserted between grouped lines.

Provide exactly one split strategy at a time.

yaml
type: io.kestra.plugin.core.storage.Split

Examples

Split a file by size.

yaml
    id: split_bytes
    namespace: company.team

    tasks:
      - id: download
        type: io.kestra.plugin.core.http.Download
        uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

      - id: split
        type: io.kestra.plugin.core.storage.Split
        from: "{{ outputs.download.uri }}"
        bytes: 5KB

Split a file by rows count.

yaml
    id: split_rows
    namespace: company.team

    tasks:
      - id: download
        type: io.kestra.plugin.core.http.Download
        uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

      - id: split
        type: io.kestra.plugin.core.storage.Split
        from: "{{ outputs.download.uri }}"
        rows: 10

Split a file in a defined number of partitions.

yaml
    id: split_partitions
    namespace: company.team

    tasks:
      - id: download
        type: io.kestra.plugin.core.http.Download
        uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

      - id: split
        type: io.kestra.plugin.core.storage.Split
        from: "{{ outputs.download.uri }}"
        partitions: 4

Split a file by regex pattern - group lines by log level.

yaml
    id: storage_tasks
    namespace: company.team

    tasks:
      - id: generate_logs
        type: io.kestra.plugin.scripts.shell.Commands
        commands:
          - echo "INFO - wow
ERROR - no
INFO - ok" > logs.txt
        outputFiles:
          - logs.txt

      - id: split
        type: io.kestra.plugin.core.storage.Split
        from: "{{ outputs.echo.outputFiles['logs.txt'] }}"
        regexPattern: "^(\w+)"

Properties

from *string

The file to be split

Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.

bytes string

Split a large file into multiple chunks with a maximum file size of bytes.

Can be provided as a string in the format "10MB" or "200KB". Must be KB or higher. This allows you to process large files, slit them into smaller chunks by lines and process them in parallel. For example, MySQL by default limits the size of a query size to 16MB per query. Trying to use a bulk insert query with input data larger than 16MB will fail. Splitting the input data into smaller chunks is a common strategy to circumvent this limitation. By dividing a large data set into chunks smaller than the max_allowed_packet size (e.g., 10MB), you can insert the data in multiple smaller queries. This approach not only helps to avoid hitting the query size limit but can also be more efficient and manageable in terms of memory utilization, especially for very large datasets. In short, by splitting the file by bytes, you can bulk-insert smaller chunks of e.g. 10MB in parallel to avoid this limitation.

partitions integerstring

Split a file into a fixed number of partitioned files. For example, if you have a file with 1000 lines and you set partitions to 10, the file will be split into 10 files with 100 lines each.

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

regexPattern string

Split file by regex pattern. Lines are grouped by the first capture group value.

A regular expression pattern with a capture group. Lines matching this pattern will be grouped by the captured value. For example, ^(\w+) will group lines by the first word extracted from the file.

rows integerstring

A number of rows per batch. The file will then be split into chunks with that maximum number of rows.

separator string

Default\n

The separator used to split a file into chunks. By default, it's a newline \n character. If you are on Windows, you might want to use \r\n instead.

Outputs

uris array

SubTypestring

The URIs of split files in Kestra's internal storage