Databricks Upload

Databricks Upload
Certified
Upload a file to DBFS

Streams a file from Kestra internal storage to DBFS using 1 MB chunks; suited for large files.

yaml
type: io.kestra.plugin.databricks.dbfs.Upload

Examples

Upload a file to the Databricks File System.

yaml
id: databricks_dbfs_upload
namespace: company.team

inputs:
  - id: file
    type: FILE
    description: File to be uploaded to DBFS

tasks:
  - id: upload_file
    type: io.kestra.plugin.databricks.dbfs.Upload
    authentication:
    token: "{{ secret('DATABRICKS_TOKEN') }}"
    host: "{{ secret('DATABRICKS_HOST') }}"
    from: "{{ inputs.file }}"
    to: /Share/myFile.parquet

Properties

from *string

Source file URI

Internal storage URI of the file to upload (kestra:// ...)

Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.

to *string

Destination DBFS path

Absolute DBFS path such as /mnt/volume/file.parquet

accountId string

Databricks account identifier

authentication

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

configFile string

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

host string

Databricks host

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

Metrics

file.size counter

The file size

Liquibase

Tasks that run Liquibase change management commands.

Data

PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

Databricks UploadCertifiedUpload a file to DBFS

Databricks Upload

More Plugins in this Category

Liquibase

PostgreSQL

Databricks Upload
Certified
Upload a file to DBFS