DocsConcepts and ExpressionsExpressionFunction

Function​Function

Functions can be called to generate content. Functions are called by their name followed by parentheses () and may have arguments.

For instance, the range function returns a list containing an arithmetic progression of integers:

twig
{% for i in range(0, 3) %}
{{ i }},
{% endfor %}

Each header below represents a built-in function.

block

The block function is used to render the contents of a block more than once. It is not to be confused with the block tag which is used to declare blocks.

The following example will render the contents of the "post" block twice; once where it was declared and again using the block function:

twig
{% block "post" %} content {% endblock %}

{{ block("post") }}

The above example will output the following:

twig
content

content

currentEachOutput

The currentEachOutput function retrieves the current output of a sibling task when using an EachSequential task.

Look at the following flow:

yaml
tasks:
  - id: each
    type: io.kestra.core.tasks.flows.EachSequential
    tasks:
      - id: first
        type: io.kestra.core.tasks.debugs.Return
        format: "{{task.id}}"
      - id: second
        type: io.kestra.core.tasks.debugs.Return
        format: "{{ outputs.first[taskrun.value].value }}"
    value: ["value 1", "value 2", "value 3"]

To retrieve the output of the first task from the second task, you need to use the special taskrun.value variable to lookup for the execution of the first task that is on the same sequential execution as the second task. And when there are multiple levels of EachSequential, you must use the special parents variable to lookup the correct execution. For example, outputs.first[parents[1].taskrun.value][parents[0].taskrun.value] for a 3-level EachSequential.

The currentEachOutput function will facilitate this by looking up the current output of the sibling task, so you don't need to use the special variables taskrun.value and parents.

The previous example can be rewritten as follow:

yaml
tasks:
  - id: each
    type: io.kestra.core.tasks.flows.EachSequential
    tasks:
      - id: first
        type: io.kestra.core.tasks.debugs.Return
        format: "{{task.id}}"
      - id: second
        type: io.kestra.core.tasks.debugs.Return
        format: "{{ currentEachOutput(outputs.first).value }}"
    value: ["value 1", "value 2", "value 3"]

And this works no matter the number of levels of EachSequential tasks used.

json

The json function will convert any string to object allowing to access its properties

twig
{{ json('[1, 2, 3]')[0] }}
{# results in: '1' #}

{{ json('{"foo": [666, 1, 2]}').foo[0] }}
{# results in: '666' #}

{{ json('{"bar": "\u0063\u0327"}').bar }}
{# results in: 'ç' #}

yaml

The yaml function will convert any string to an object allowing to access its properties.

twig
"{{ yaml('foo: [666, 1, 2]').foo[0] }}"
{# results in: '666' #}

{{ yaml(yaml_object) | jq(...) | yaml }}
{# prints the yaml_object as a yaml string #}

max

The max function will return the largest of it's numerical arguments.

twig
{{ max(user.age, 80) }}

min

The min function will return the smallest of it's numerical arguments.

twig
{{ min(user.age, 80) }}

now

The now function will return the actual datetime. The arguments are the same as the date filter except the format is different.

twig
{{ now() }}
{{ now(timeZone="Europe/Paris") }}

Arguments

  • existingFormat
  • timeZone
  • locale

parent

The parent function is used inside of a block to render the content that the parent template would have rendered inside of the block had the current template not overridden it. It is similar to Java's super keyword.

Let's assume you have a template, "parent.peb" that looks something like this:

twig
{% block "content" %}
    parent contents
{% endblock %}

And then you have another template, "child.peb" that extends "parent.peb":

twig
{% extends "parent.peb" %}

{% block "content" %}
    child contents
    {{ parent() }}
{% endblock %}

The output will look something like the following:

twig
parent contents
child contents

range

The range function will return a list containing an arithmetic progression of numbers:

twig
{% for i in range(0, 3) %}
    {{ i }},
{% endfor %}

{# outputs 0, 1, 2, 3, #}

When step is given (as the third parameter), it specifies the increment (or decrement):

twig
{% for i in range(0, 6, 2) %}
    {{ i }},
{% endfor %}

{# outputs 0, 2, 4, 6, #}

Pebble built-in .. operator is just a shortcut for the range function with a step of 1+

twig
{% for i in 0..3 %}
    {{ i }},
{% endfor %}

{# outputs 0, 1, 2, 3, #}

read

Read an internal storage file and return its content as a string. This function accepts one of the following:

  1. A path to a Namespace File e.g. {{ read('myscript.py') }}
  2. An internal storage URI e.g. {{ read(inputs.myfile) }} or {{ read(outputs.extract.uri) }}.

Reading namespace files is restricted to files in the same namespace as the flow using this function.

Reading internal storage files is restricted to the current execution. Specifically, those are files created by the current flow's execution or the parent flow execution (for flows triggered by a Subflow task or a ForEachItem task).

twig
# Read a namespace file from the path `subdir/file.txt`
{{ read('subdir/file.txt') }}

# Read an internal storage file from the `uri` output of the `readFile` task
{{ read(outputs.readFile.uri) }}

# Read an internal storage file from an input named `file`
{{ read(inputs.file) }}

render

By default, kestra renders all expressions only once. This is safer from a security perspective, and it prevents unintended behavior when parsing JSON elements of a webhook payload that contained a templated string from other applications (such as GitHub Actions or dbt core). However, sometimes recursive rendering is desirable. For example, if you want to parse flow variables that contain Pebble expressions. This is where the render() function comes in handy.

The render() function is used to enable recursive rendering of Pebble expressions. It is available since the release 0.14.0.

The syntax for the render() function is as follows:

yaml
{{ render(expression_string, recursive=true) }} # if false, render only once

The function takes two arguments:

  1. The string of an expression to be rendered e.g. "{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}"
  2. A boolean flag that controls whether the rendering should be recursive or not:
    • if true (default), the expression will be rendered recursively
    • if false, the expression will be rendered only once.

Let's see some examples of how the render() function works and where you need to use it.

Where the render() function is not needed

Let's take the following flow as an example:

yaml
id: parse_input_and_trigger_expressions
namespace: qa

inputs:
  - id: myinput
    type: STRING
    defaults: hello

tasks:
  - id: parse_date
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}"

  - id: parse_input
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ inputs.myinput }}"

triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "* * * * *"

Since we don't use any nested expressions (like using trigger or input expressions in variables, or using variables in other variables), this flow will work just fine without having to use the render() function.

Where the render() function is needed

Let's modify our example so that now we store that long expression "{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}" in a variable:

yaml
id: hello
namespace: dev

variables:
  trigger_var: "{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}"

tasks:
  - id: parse_date
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ vars.trigger_var }}" # no render function means no recursive rendering for that trigger_var variable

triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    disabled: true
    cron: "* * * * *"

Here, the task parse_date will print the expression without recursively rendering it, so the printed output will be a string of that variable rather than a parsed date:

shell
{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}

To fix that, simply wrap the vars.trigger_var variable in the render() function:

yaml
id: hello
namespace: dev

variables:
  trigger_var: "{{ trigger.date ?? execution.startDate | date('yyyy-MM-dd') }}"

tasks:
  - id: parse_date
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ render(vars.trigger_var) }}" # this will print the recursively-rendered expression

triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "* * * * *"

Now you should see the date printed in the logs, e.g. 2024-01-16.

Using expressions in namespace variables

Let's assume that you have configured a namespace variable myvar. That variable uses a Pebble function to retrieve a secret e.g. {{ secret('MY_SECRET') }}. By default, kestra will only render the expression without recursively rendering what's inside of the namespace variable:

yaml
id: namespace_variable
namespace: qa

tasks:
  - id: print_variable
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ namespace.myvar }}"

If you want to render the secret contained in that variable, you will need to wrap it in the render() function:

yaml
id: namespaces_variable
namespace: qa

tasks:
  - id: print_variable
    type: io.kestra.core.tasks.debugs.Return
    format: "{{ render(namespace.myvar) }}"

String concatenation

Let's look at another example that will demonstrate how the render() function works with string concatenation.

yaml
id: pebble_string_concatenation
namespace: qa

inputs:
  - id: date
    type: DATETIME
    defaults: 2024-02-24T22:00:00.000Z

  - id: user
    type: STRING
    defaults: Rick

variables:
  day_of_week: "{{ trigger.date ?? inputs.date | date('EEEE') }}"
  full_date: "{{ vars.day_of_week }}, the {{ trigger.date ?? inputs.date | date('yyyy-MM-dd') }}"
  full_date_concat: "{{ vars.day_of_week ~ ', the ' ~ (trigger.date ?? inputs.date | date('yyyy-MM-dd')) }}"
  greeting_concat: "{{'Hello, ' ~ inputs.user ~ ' on ' ~ vars.full_date }}"
  greeting_brackets: "Hello, {{ inputs.user }} on {{ vars.full_date }}"

tasks:
  - id: not-rendered
    type: io.kestra.core.tasks.log.Log
    message: |
     Concat: {{ vars.greeting_concat }}
     Brackets: {{ vars.greeting_brackets }}
     Full date: {{ vars.full_date }}
     Full date concat: {{ vars.full_date_concat }}

  - id: rendered-recursively
    type: io.kestra.core.tasks.log.Log
    message: |
     Concat: {{ render(vars.greeting_concat) }}
     Brackets: {{ render(vars.greeting_brackets) }}
     Full date: {{ render(vars.full_date) }}
     Full date concat: {{ render(vars.full_date_concat) }}

  - id: rendered-once
    type: io.kestra.core.tasks.log.Log
    message: |
     Concat: {{ render(vars.greeting_concat, recursive=false) }}
     Brackets: {{ render(vars.greeting_brackets, recursive=false) }}
     Full date: {{ render(vars.full_date, recursive=false) }}
     Full date concat: {{ render(vars.full_date_concat, recursive=false) }}

triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: "* * * * *"

Note that:

  • the ?? syntax within "{{ trigger.date ?? inputs.date | date('EEEE') }}" is a null-coalescing operator that returns the first non-null value in the expression. For example, if trigger.date is null, the expression will return inputs.date.
  • the ~ sign is a string concatenation operator that combines two strings into one.

When you run this flow, you should see the following output in the logs:

shell
INFO Concat: {{'Hello, ' ~ inputs.user ~ ' on ' ~ vars.full_date }}
Brackets: Hello, {{ inputs.user }} on {{ vars.full_date }}
Full date: {{ vars.day_of_week }}, the {{ trigger.date ?? inputs.date | date('yyyy-MM-dd') }}
Full date concat: {{ vars.day_of_week ~ ', the ' ~ (trigger.date ?? inputs.date | date('yyyy-MM-dd')) }}

INFO Concat: Hello, Rick on Saturday, the 2024-02-24
Brackets: Hello, Rick on Saturday, the 2024-02-24
Full date: Saturday, the 2024-02-24
Full date concat: Saturday, the 2024-02-24

INFO Concat: Hello, Rick on {{ vars.day_of_week }}, the {{ trigger.date ?? inputs.date | date('yyyy-MM-dd') }}
Brackets: Hello, Rick on {{ vars.day_of_week }}, the {{ trigger.date ?? inputs.date | date('yyyy-MM-dd') }}
Full date: {{ trigger.date ?? inputs.date | date('EEEE') }}, the 2024-02-24
Full date concat: {{ trigger.date ?? inputs.date | date('EEEE') }}, the 2024-02-24

You may notice that both the vars.greeting_concat and vars.greeting_brackets lead to the same output, even though the first one uses the ~ sign for string concatenation within a single Pebble expression {{ }}, and the second one uses one string with multiple {{ }} expressions to concatenate the strings. Both are fully supported and you can decide which one to use based on your preference.

Webhook trigger: using render() with the recursive=false flag

Let's look at how the render() function works with the webhook trigger.

Imagine you use a GitHub webhook as a trigger in order to automate deployments after a pull request is merged. Your GitHub webhook payload looks as follows:

json
{
    "pull_request": {
        "html_url": "https://github.com/kestra-io/kestra/pull/2834",
        "body": "This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}."
    }
}

The pull request body contains templated variables ${{ env.MYENV }} and ${{ secrets.GITHUB_TOKEN }}, which are not meant to be rendered by Kestra, but by GitHub Actions. Processing this webhook payload with recursive rendering would result in an error, as those variables are not defined in the flow execution context.

In order to retrieve elements such as the pull_request.body from that webhook's payload without recursively rendering its content, you can leverage the render() function with the recursive=false flag:

yaml
id: pebble_in_webhook
namespace: qa

inputs:
  - id: github_url
    type: STRING
    defaults: https://github.com/kestra-io/kestra/pull/2834

  - id: body
    type: JSON
    defaults: |
      {
        "pull_request": {
            "html_url": "https://github.com/kestra-io/kestra/pull/2834",
            "body": "This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}"
        }
      }

variables:
  body: "{{ trigger.body.pull_request.body ?? trigger.body.issue.body ?? inputs.body }}"
  github_url: "{{ trigger.body.pull_request.html_url ?? trigger.body.issue.html_url ?? inputs.github_url }}"

tasks:
  - id: render_once
    type: io.kestra.core.tasks.log.Log
    message: "{{ render(vars.body, recursive=false) }}"

  - id: not_recursive
    type: io.kestra.core.tasks.log.Log
    message: "{{ vars.body }}"

  - id: recursive
    type: io.kestra.core.tasks.log.Log
    message: "{{ render(vars.body) }}"
    allowFailure: true # this task will fail because it will try to render the webhook's payload

triggers:
  - id: webhook
    type: io.kestra.core.models.triggers.types.Webhook
    key: test1234

Only the render_once task is relevant for this use case, as it will render the pull request's body without recursively rendering its content. The flow includes a recursive and non-recursive configuration for easy comparison.

  • The not_recursive task will print the {{ trigger.body.pull_request.body ?? trigger.body.issue.body ?? inputs.body }} expression as a string without rendering it.
  • The recursive task will fail, as it will try to render the webhook's payload containing templating that cannot be parsed by kestra.

Here is how you can call that flow via curl:

shell
curl -i -X POST -H "Content-Type: application/json" \
  -d '{ "pull_request": {"html_url": "https://github.com/kestra-io/kestra/pull/2834", "body": "This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}"} }' \
  http://localhost:8080/api/v1/executions/webhook/qa/pebble_in_webhook/test1234

On an instance with multi-tenancy enabled, make sure to also pass the tenant ID in the URL:

shell
curl -i -X POST -H "Content-Type: application/json" \
  -d '{ "pull_request": {"html_url": "https://github.com/kestra-io/kestra/pull/2834"}, "body": "This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}"} }' \
  http://localhost:8080/api/v1/your_tenant/executions/webhook/qa/pebble_in_webhook/test1234

You should see similar output in the logs:

shell
INFO This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}
INFO {{ trigger.body.pull_request.body ?? trigger.body.issue.body ?? inputs.body }}
ERROR io.pebbletemplates.pebble.error.PebbleException: Missing variable: 'env' on 'This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}' at line 1 (?:?)
ERROR Missing variable: 'env' on 'This PR replaces the ${{ env.GITHUB_TOKEN }} with a more secure ${{ secrets.GITHUB_TOKEN }}' at line 1 (?:?)

renderOnce

The renderOnce() function is used to enable one-time rendering of nested Pebble expressions. It is available since the release 0.16.0 and is equivalent to render(expression_string, recursive=false).

This function is syntactic sugar to reduce overhead brought by the recursive rendering default behaviour. It can be used to use vars easily as they may contain themselves pebble expressions.

Basically, if vars.a={{ vars.b }}, vars.b=42 then renderOnce(vars.a)=42. Note that if vars.b={{ vars.c }}, renderOnce(vars.a)={{ vars.c }}.

The syntax for the renderOnce() function is as follows:

yaml
{{ renderOnce(expression_string) }}

secret

The secret() function is used to retrieve a secret from a secret backend based on the key provided as input to that function.

Here is an example flow that retrieves the Personal Access Token secret stored using the secret key GITHUB_ACCESS_TOKEN:

yaml
id: secret
namespace: dev

tasks:
  - id: githubPAT
    type: io.kestra.core.tasks.log.Log
    message: "{{ secret('GITHUB_ACCESS_TOKEN') }}"

The secret('key') function will lookup up the configured secret manager backend for a secret value using the key GITHUB_ACCESS_TOKEN. If the key is missing, an exception will be raised. The example flow shown above will look up the secret value by key and will log the output with the secret value.

There are some differences between the secret management backend in the Open-Source and Enterprise editions. By default, Kestra provides a secret management backend based on environment variables. Each environment variable starting with SECRET_ will be available as a secret, and its value must be base64-encoded.

The above example will:

  1. retrieve the secret GITHUB_ACCESS_TOKEN from an environment variable SECRET_GITHUB_ACCESS_TOKEN
  2. base64-decode it at runtime.