Storage configuration
Kestra needs an internal storage to store data processed by tasks. This includes files from flow inputs and data stored as task outputs.
The default internal storage implementation is the local storage which is not suitable for production as it will store data inside a local folder on the host filesystem.
This local storage can be configured as follows:
kestra:
storage:
type: local
local:
base-path: /tmp/kestra/storage/ # your custom path
Other internal storage types include:
- Storage S3 for AWS S3
- Storage GCS for Google Cloud Storage
- Storage Minio compatible with others S3 like storage services
- Storage Azure for Azure Blob Storage
S3
First, make sure that the S3 storage plugin is installed in your environment. You can install it with the following Kestra command:
./kestra plugins install io.kestra.storage:storage-s3:LATEST
. This command will download the plugin's jar file into the plugin's directory.
Then, enable the storage using the following configuration:
kestra:
storage:
type: s3
s3:
accessKey: "<your-aws-access-key-id>"
secretKey: "<your-aws-secret-access-key>"
region: "<your-aws-region>"
bucket: "<your-s3-bucket-name>"
If you are using an AWS EC2 or EKS instance, you can use the default credentials provider chain. In this case, you can omit the accessKey
and secretKey
options:
kestra:
storage:
type: s3
s3:
region: "<your-aws-region>"
bucket: "<your-s3-bucket-name>"
Minio
If you use Minio or similar S3-compatible storage options, you can follow the same process as shown above to install the Minio storage plugin. Then, make sure to include the Minio's endpoint
and port
in the storage configuration:
kestra:
storage:
type: minio
minio:
endpoint: "<your-endpoint>"
port: "<your-port>"
secure: "<your-secure>"
accessKey: "<your-accessKey>"
secretKey: "<your-secretKey>"
region: "<your-region>"
bucket: "<your-bucket>"
Azure
First, install the Azure storage plugin. To do that, you can leverage the following Kestra command:
./kestra plugins install io.kestra.storage:storage-azure:LATEST
. This command will download the plugin's jar file into the plugins directory.
Adjust the storage configuration shown below depending on your chosen authentication method:
kestra:
storage:
type: azure
azure:
endpoint: "https://unittestkt.blob.core.windows.net"
container: storage
connection-string: "<connection-string>"
shared-key-account-name: "<shared-key-account-name>"
shared-key-account-access-key: "<shared-key-account-access-key>"
sas-token: "<sas-token>"
GCS
You can install the GCS storage plugin using the following Kestra command:
./kestra plugins install io.kestra.storage:storage-gcs:LATEST
. This command will download the plugin's jar file into the plugins directory.
Then, you can enable the storage using the following configuration:
kestra:
storage:
type: gcs
gcs:
bucket: "<your-bucket-name>"
service-account: "<service-account key as JSON or use default credentials>"
project-id: "<project-id or use default projectId>"
If you haven't configured the kestra.storage.gcs.service-account
option, Kestra will use the default service account, which is:
- the service account defined on the cluster (for GKE deployments)
- the service account defined on the compute instance (for GCE deployments).
You can also provide the environment variable GOOGLE_APPLICATION_CREDENTIALS
with a path to a JSON file containing GCP service account key.
You can find more details in the GCP documentation.