Working with Yaml

Most of Kestra resource need to be described as Yaml like kestra_flow & kestra_template.

We have chosen to use a full yaml in terraform definition since the structure is recursive and dynamic, so it can't be described using terraform internal schema.

There is 2 ways (for flow) to handle yaml:

• use keep_original_source = true method: the default one, the raw yaml will be send and save in Kestra.
• use keep_original_source = false method: the yaml will be encoded in json before behind to the server, so comment and indent will be handle by the server

Those properties have to be set at the provider level.

There is in terraform a lot of function that allow to work properly with this yaml content :

Simple multiline string example

You can use simple terraform multiline string with Heredoc String :

resource "kestra_flow" "example" {
  namespace = "io.kestra.mynamespace"
  flow_id = "my-flow"
  content = <<EOT
inputs:
  - id: my-value
    type: STRING
    required: true

tasks:
  - id: t2
    type: io.kestra.core.tasks.log.Log
    message: first {{task.id}}
    level: TRACE
EOT
}

External files

Better will be to use a file function. Just create a file .yml near your terraform .tf and include the whole file in your resource:

inputs:
  - id: my-value
    type: STRING
    required: true

tasks:
  - id: t2
    type: io.kestra.core.tasks.log.Log
    message: first {{task.id}}
    level: TRACE
EOT

resource "kestra_flow" "example" {
  namespace = "io.kestra.mynamespace"
  flow_id = "my-flow"
  content = file("my-flow.yml")
}

External files with template

Even better will be to use a templatefile function that will allow more complex flows to be more readable. You can include some external external and this one can also include other file.

Dealing with included yaml string

Imagine a flow that will query an external database. Embedding the full query can lead to very long flow definition. In the case you can use templatefile to allow inclusion of an external files from the yaml.

Create a sql file:

SELECT *
FROM ....

Create the yaml file for the flow:

tasks:
  - id: "query"
    type: "io.kestra.plugin.jdbc.mysql.Query"
    url: jdbc:postgresql://127.0.0.1:56982/
    username: postgres
    password: pg_passwd
    sql: |
      ${indent(6, file("my-query.sql"))}
    fetchOne: true

And finally create the resource invoking the templatefile:

resource "kestra_flow" "example" {
  namespace = "io.kestra.mynamespace"
  flow_id = "my-flow"
  content = templatefile("my-flow.yaml", {})
}

The tf files will required the yaml files that will require the sql files and the final flow will be:

tasks:
  - id: "query"
    type: "io.kestra.plugin.jdbc.mysql.Query"
    url: jdbc:postgresql://127.0.0.1:56982/
    username: postgres
    password: pg_passwd
    sql: |
      SELECT *
      FROM ....
    fetchOne: true

Include full yaml part

By the same way, you can also include a full yaml specs inside another one.

Create 2 yaml files:

id: t1
type: io.kestra.core.tasks.log.Log
message: first {{task.id}}
level: TRACE

id: t2
type: io.kestra.core.tasks.log.Log
message: second {{task.id}}
level: TRACE

Create the yaml file for the flow:

tasks:
  - ${indent(4, file("t1.yml"))}
  - ${indent(4, file("t2.yml"))}