Compression ArchiveCompress

Compression ArchiveCompress
Certified
Create an archive from multiple files or a directory

Builds an archive from rendered file map inputs stored in internal storage and/or from a whole directory walked recursively with relative paths preserved, optionally wrapping it with a stream compressor (for example TAR + GZIP). Fails for algorithms that are extract-only.

yaml
type: io.kestra.plugin.compress.ArchiveCompress

Examples

Compress an input file

yaml
id: archive_compress
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: "archive_compress"
    type: "io.kestra.plugin.compress.ArchiveCompress"
    from:
      myfile.txt: "{{ inputs.file }}"
    algorithm: ZIP

Download two files, compress them together and upload to S3 bucket

yaml
id: archive_compress
namespace: company.team

tasks:
  - id: products_download
    type: io.kestra.plugin.core.http.Download
    uri: "http://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv"

  - id: orders_download
    type: io.kestra.plugin.core.http.Download
    uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"

  - id: archive_compress
    type: "io.kestra.plugin.compress.ArchiveCompress"
    from:
      products.csv: "{{ outputs.products_download.uri }}"
      orders.csv: "{{ outputs.orders_download.uri }}"
    algorithm: TAR
    compression: GZIP

  - id: upload_compressed
    type: io.kestra.plugin.aws.s3.Upload
    bucket: "example"
    region: "{{ secret('AWS_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    from: "{{ outputs.archive_compress.uri }}"
    key: "archive.gz"

Compress a whole namespace directory, preserving relative paths

yaml
id: archive_compress_directory
namespace: company.team

tasks:
  - id: archive_compress
    type: "io.kestra.plugin.compress.ArchiveCompress"
    fromDirectory: "nsfile:///reports"
    algorithm: ZIP

Compress a directory of the task working directory

yaml
id: archive_compress_working_directory
namespace: company.team

tasks:
  - id: working_dir
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: generate
        type: io.kestra.plugin.scripts.shell.Commands
        taskRunner:
          type: io.kestra.plugin.core.runner.Process
        commands:
          - mkdir -p data/nested && echo hello > data/nested/a.txt

      - id: archive_compress
        type: "io.kestra.plugin.compress.ArchiveCompress"
        fromDirectory: "data"
        algorithm: TAR
        compression: GZIP

Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-compress --compatible-for $KESTRA_VERSION

For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version. For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

algorithm *string

Possible Values

ARARJCPIODUMPJARTARZIP

Archive container format to create

Required archive format. Compression supports AR, CPIO, JAR, TAR, and ZIP; ARJ and DUMP are extract-only and will fail on compression.

assets

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions

io.kestra.core.models.assets.AssetsDeclaration

assetFailureBehaviorstring

Possible Values

IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

io.kestra.core.models.assets.AssetIdentifier

id*string

Min length1

typestring

outputs

The assets produced as outputs.

io.kestra.plugin.ee.assets.Dataset

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.File

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.Table

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.VM

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.core.models.assets.External

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.core.models.assets.Custom

id*string

Min length1

Max length150

type*string

Min length1

Custom asset type

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

compression string

Possible Values

BROTLIBZIP2DEFLATEDEFLATE64GZIPLZ4BLOCKLZ4FRAMELZMASNAPPYSNAPPYFRAMEXZZZSTD

Optional compressor applied to the archive stream

Use a single-file compressor such as GZIP alongside TAR. Leave null to store the archive uncompressed. Brotli, Deflate64, and Snappy variants are decode-only and will fail during compression.

from object

Structured data items, either as a map, a list of map, a URI, or a JSON string.

Structured data items can be defined in the following ways:

A single item as a map (a document).
A list of items as a list of maps (a list of documents).
A URI, supported schemes are kestra for internal storage files, file for host local files, and nsfile for namespace files.
A JSON String that will then be serialized either as a single item or a list of items. Optional when fromDirectory is set.

fromDirectory string

Directory to archive recursively

Every file below this directory is added to the archive with its path relative to the directory root, and directories (including empty ones) are added as entries, except for the AR format which has no directory support. Supported locations:

a namespace directory, e.g. nsfile:///path/to/dir (current namespace) or nsfile://my.namespace/path/to/dir;
a namespace directory given as its internal storage URI, e.g. kestra:///my/namespace/_files/path/to/dir, handled like nsfile://;
any other directory of Kestra's internal storage, e.g. a kestra:///... execution output directory, whose stored files are archived as they are; namespace metadata directories (_files, _kv, ...) met at the namespace level are skipped;
a path inside the task working directory, e.g. my/dir or . (symbolic links are skipped, the archive being created is never included). The directory must exist. Can be combined with from: the directory content is written first, then the from entries, and a from entry named like a file of the directory is rejected.