Variables

Variables are key-value pairs that help reuse some values across tasks.

You can also store variables on a namespace level so that they can be reused across multiple flows in a given namespace.

How to configure variables

The example below shows how you can configure variables in your flow:

yaml

id: hello_world
namespace: company.team

variables:
  myvar: hello
  numeric_variable: 42

tasks:
  - id: log
    type: io.kestra.plugin.core.debug.Return
    format: "{{ vars.myvar }} world {{ vars.numeric_variable }}"

You can see the syntax for using variables is {{ vars.variable_name }}.

How are variables rendered

You can use variables in any task property that is documented as dynamic.

Dynamic variables are rendered thanks to the Pebble templating engine. Pebble templating engine processes various expressions with filters and functions. More information on variable processing can be found under Expressions.

Since 0.14, Variables are no longer rendered recursively. You can read more about this change and how to change this behaviour in the 0.14 migration guide.

Dynamic Variables

If you want to have an expression inside of your variable, you need to wrap it in render when you use it in a task.

For example, this variable only displays the current time in the log message when wrapped in render. Otherwise, the log message just contains the expression as a string:

yaml

id: dynamic_variable
namespace: company.team

variables:
  time: "{{ now() }}"

tasks:
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{ render(vars.time) }}"

You need to wrap the variable expression in render every time you want to use it in a task.

FAQ

How do I escape a block in Pebble syntax to ensure that it won't be parsed?

To ensure that a block of code won't be parsed by Pebble, you can use the {% raw %} and {% endraw %} Pebble tags. For example, the following Pebble expression returns the string {{ myvar }} instead of the value of the myvar variable:

yaml

{% raw %}{{ myvar }}{% endraw %}

Which order are inputs and variables resolved?

Inputs are resolved first, even before the execution starts. In fact, if you try to create a flow with an invalid input value, the execution will not be created.

Therefore, you can use inputs within variables, but you can't use variables or Pebble expressions within inputs.

Expressions are rendered recursively, meaning that if a variable contains another variable, the inner variable will be resolved first.

When it comes to triggers, they are handled similarly to inputs as they are known before the execution starts (they trigger the execution). This means that you can't use inputs (unless they have defaults attached) or variables within triggers, but you can use trigger variables within variables.

To make it clearer, let's look at some examples.

Examples

This flow uses inputs, trigger, and execution variables which are resolved before variables:

yaml

id: upload_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: declarative-data-orchestration

tasks:
  - id: get_zip_file
    type: io.kestra.plugin.core.http.Download
    uri: https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip

  - id: unzip
    type: io.kestra.plugin.compress.ArchiveDecompress
    algorithm: ZIP
    from: "{{outputs.get_zip_file.uri}}"

  - id: csv_upload
    type: io.kestra.plugin.aws.s3.Upload
    from: "{{ outputs.unzip.files['global_power_plant_database.csv'] }}"
    bucket: "{{ inputs.bucket }}"
    key: "powerplant/{{ trigger.date ?? execution.startDate | date('yyyy_MM_dd__HH_mm_ss') }}.csv"

triggers:
  - id: hourly
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "@hourly"

This flow starts a task conditionally based on whether the input is provided or not:

yaml

id: conditional_branching
namespace: company.team

inputs:
  - id: parameter
    type: STRING
    required: false

tasks:
  - id: if
    type: io.kestra.plugin.core.flow.If
    condition: "{{inputs.customInput ?? false }}"
    then:
      - id: if_not_null
        type: io.kestra.plugin.core.log.Log
        message: Received input {{inputs.parameter}}
    else:
      - id: if_null
        type: io.kestra.plugin.core.log.Log
        message: No input provided

Below is an example that uses a trigger variable within a trigger itself (that's allowed!):

yaml

id: backfill_past_mondays
namespace: company.team

tasks:
  - id: log_trigger_or_execution_date
    type: io.kestra.plugin.core.log.Log
    message: "{{ trigger.date ?? execution.startDate }}"

triggers:
  - id: first_monday_of_the_month
    type: io.kestra.plugin.core.trigger.Schedule
    timezone: Europe/Berlin
    backfill:
      start: 2023-11-11T00:00:00Z
    cron: "0 11 * * MON" # at 11 on every Monday
    conditions: # only first Monday of the month
      - type: io.kestra.plugin.core.condition.DayWeekInMonth
        date: "{{ trigger.date }}"
        dayOfWeek: "MONDAY"
        dayInMonth: "FIRST"

Can I transform variables with Pebble expressions?

Yes. Kestra uses Pebble Templates along with the execution context to render dynamic properties. This means that you can use Pebble expressions (such as filters, functions, and operators to transform inputs and variables.

The example below illustrates how to use variables and Pebble expressions to transform string values in dynamic task properties:

yaml

id: variables_demo
namespace: company.team

variables:
  DATE_FORMAT: "yyyy-MM-dd"

tasks:
  - id: seconds_of_day
    type: io.kestra.plugin.core.debug.Return
    format: '{{60 * 60 * 24}}'

  - id: start_date
    type: io.kestra.plugin.core.debug.Return
    format: "{{ execution.startDate | date(vars.DATE_FORMAT) }}"

  - id: curr_date_unix
    type: io.kestra.plugin.core.debug.Return
    format: "{{ now() | date(vars.DATE_FORMAT) | timestamp() }}"

  - id: next_date
    type: io.kestra.plugin.core.debug.Return
    format: "{{ now() | dateAdd(1, 'DAYS') | date(vars.DATE_FORMAT) }}"

  - id: next_date_unix
    type: io.kestra.plugin.core.debug.Return
    format: "{{ now() | dateAdd(1, 'DAYS') | date(vars.DATE_FORMAT) | timestamp() }}"

  - id: pass_downstream
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - echo {{outputs.next_date_unix.value}}

Can I use nested variables?

Yes! However, keep in mind that depending on the task, you may need to wrap the root variable in a json() function in order to access specific keys. Below is an example using a list of maps as a variable:

yaml

id: vars
namespace: company.myteam

variables:
  servers:
    - fqn: server01.mydomain.io
      user: root
    - fqn: server02.mydomain.io
      user: guest
    - fqn: server03.mydomain.io
      user: rick

tasks:
  - id: parallel
    type: io.kestra.plugin.core.flow.ForEach
    concurrencyLimit: 0
    values: "{{ vars.servers }}"
    tasks:
      - id: log
        type: io.kestra.plugin.core.log.Log
        message:
           - "{{ taskrun.value }}" # for each element in the servers list, this will print the full JSON object e.g. {“fqn”:“server01.mydomain.io”,“user”:“root”}
           - "{{ json(taskrun.value).fqn }}" # prints the value for that key e.g. server01.mydomain.io
           - "{{ json(taskrun.value).user }}" # prints the value for that key e.g. root

Was this page helpful?

Workflow ComponentsExecution

Workflow ComponentsInputs

​Variables

Variables