Databricks Upload
CertifiedUpload a file to DBFS
Databricks Upload
Upload a file to DBFS
Streams a file from Kestra internal storage to DBFS; suited for large files.
type: io.kestra.plugin.databricks.dbfs.UploadExamples
Upload a file to the Databricks File System.
id: databricks_dbfs_upload
namespace: company.team
inputs:
- id: file
type: FILE
description: File to be uploaded to DBFS
tasks:
- id: upload_file
type: io.kestra.plugin.databricks.dbfs.Upload
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
host: "{{ secret('DATABRICKS_HOST') }}"
from: "{{ inputs.file }}"
to: /Share/myFile.parquet
Properties
from *string
Source file URI
Internal storage URI of the file to upload (kestra:// ...)
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
to *string
Destination DBFS path
Absolute DBFS path such as /mnt/volume/file.parquet
accountId string
Databricks account identifier
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150authentication
Databricks authentication configuration
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
Authentication type
Azure client ID
Azure client secret
Azure tenant ID
Client ID
Client secret
Google credentials JSON
Google service account email
Password
Databricks personal access token
Username
configFile string
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one
host string
Databricks host
Metrics
file.size counter
The file size
