Unity Catalog - Volumes Upload
CertifiedUpload a file to a Unity Catalog volume
Unity Catalog - Volumes Upload
Upload a file to a Unity Catalog volume
Streams a file from Kestra internal storage to a Unity Catalog volume using the Databricks Files API. The volume must already exist. This is the replacement for the legacy DBFS upload.
type: io.kestra.plugin.databricks.unitycatalog.volume.UploadExamples
Upload a file to a Unity Catalog volume
id: databricks_uc_volume_upload
namespace: company.team
inputs:
- id: file
type: FILE
description: File to upload to the volume
tasks:
- id: upload_file
type: io.kestra.plugin.databricks.unitycatalog.volume.Upload
host: "{{ secret('DATABRICKS_HOST') }}"
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
from: "{{ inputs.file }}"
volumePath: /Volumes/main/landing_zone/raw_files/data.csv
overwrite: true
Properties
from *string
Source file URI
Internal storage URI of the file to upload (kestra://...).
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
volumePath *string
Volume file path
Absolute path of the destination file, in the form /Volumes/<catalog>/<schema>/<volume>/<path>.
accountId string
Databricks account identifier
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150authentication
Databricks authentication configuration
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
Authentication type
Azure client ID
Azure client secret
Azure tenant ID
Client ID
Client secret
Google credentials JSON
Google service account email
Password
Databricks personal access token
Username
configFile string
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one
host string
Databricks host
overwrite booleanstring
Overwrite existing file
Replace the file if it already exists. Defaults to false, in which case the upload fails when the file exists.
Outputs
size integer
Size of the uploaded file, in bytes
volumePath string
Path of the uploaded file in the volume
Metrics
file.size counter
The size of the uploaded file, in bytes
