Kestra disabled trigger reenable guard

id: kestra-disabled-trigger-reenable-guard
namespace: company.team

description: |
  Find Kestra triggers that an operator declared should be live but that are switched off, the
  schedules nobody turned back on after an incident. List them in dry run. On auto_remediate,
  behind a Pause approval, switch each one back on with the Kestra plugin, then read the trigger
  state again and fail if any targeted trigger is still off. It only ever acts on the
  desired_live allowlist, so a trigger somebody paused on purpose is reported, never restarted.

inputs:
  - id: kestra_url
    type: STRING
    displayName: Target Kestra API URL
    description: The Kestra whose triggers are checked. The default points the flow
      at its own cluster.
    defaults: "http://localhost:8080"

  - id: namespace
    type: STRING
    displayName: Namespace to check
    description: Only triggers in exactly this namespace are considered. A safe
      narrow scope.
    defaults: company.team

  - id: desired_live
    type: ARRAY
    itemType: STRING
    displayName: Triggers that must be live
    description: >-
      The allowlist, one entry per trigger as `flowId/triggerId`. A trigger is
      switched back on only if it appears here. Anything off that is not listed
      is reported and left alone. Empty means the flow reports everything it
      finds without changing a thing.
    defaults: []

  - id: max_enables
    type: INT
    displayName: Enable ceiling
    description: Refuse to switch on more than this many triggers in one run. A
      larger number means something switched off a whole namespace, which is an
      incident for a human.
    defaults: 10

  - id: kestra_user
    type: STRING
    displayName: Kestra API username
    defaults: admin@kestra.io

  - id: kestra_password
    type: STRING
    displayName: Kestra API password
    defaults: "{{ secret('KESTRA_PASSWORD') }}"

  - id: auto_remediate
    type: BOOL
    displayName: Auto remediate
    description: Allow the switch back on. Still requires dry_run false and a Pause
      approval.
    defaults: false

  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: Must be false, together with auto_remediate, before any trigger is
      switched on.
    defaults: true

variables:
  allow_json: "{{ inputs.desired_live | jq('.') | first | toJson }}"
  select_program: "{{ render(vars.allow_json) ~ ' as $allow | [.results[] | {flow:
    .state.flowId, trig: .state.triggerId, key: (.state.flowId + \"/\" +
    .state.triggerId), off_state: (.state.disabled == true), off_source:
    (.state.sourceDisabled == true), next: (.state.nextEvaluationDate //
    \"never\"), last: (.state.lastTriggeredDate // \"never\")}] |
    map(select(.off_state or .off_source)) as $off | {targets: ($off |
    map(select(([.key] - $allow) | length == 0)) | map(select(.off_state))),
    blocked: ($off | map(select(([.key] - $allow) | length == 0)) |
    map(select(.off_state == false))), ignored: ($off | map(select(([.key] -
    $allow) | length > 0)))}' }}"

tasks:
  - id: detect
    type: io.kestra.plugin.core.http.Request
    description: Read-only. Ask the target Kestra for every trigger in the
      namespace, with both the stored enabled flag and the flag that comes from
      the flow source.
    uri: "{{ inputs.kestra_url }}/api/v1/main/triggers/search"
    method: GET
    contentType: application/json
    options:
      auth:
        type: BASIC
        username: "{{ inputs.kestra_user }}"
        password: "{{ inputs.kestra_password }}"
    params:
      filters[namespace][EQUALS]: "{{ inputs.namespace }}"
      size: "500"

  - id: select
    type: io.kestra.plugin.core.output.OutputValues
    description: Split every switched-off trigger into three sets, the allowlisted
      ones this flow can switch on, the allowlisted ones that are off in the
      flow source, plus everything off that is not on the allowlist.
    values:
      report: "{{ outputs.detect.body | jq(render(vars.select_program)) | first |
        toJson }}"

  - id: counts
    type: io.kestra.plugin.core.output.OutputValues
    values:
      target_count: "{{ outputs.select.values.report | jq('.targets | length') | first }}"
      target_keys: "{{ outputs.select.values.report | jq('[.targets[] | {f: .flow, t:
        .trig}]') | first | toJson }}"
      target_key_list: "{{ outputs.select.values.report | jq('[.targets[].key]') |
        first | toJson }}"
      target_detail: "{{ outputs.select.values.report | jq('[.targets[] | .key + \"
        (last fired \" + .last + \")\"] | join(\"; \")') | first }}"
      blocked_count: "{{ outputs.select.values.report | jq('.blocked | length') | first }}"
      blocked_detail: "{{ outputs.select.values.report | jq('[.blocked[].key] |
        join(\"; \")') | first }}"
      ignored_count: "{{ outputs.select.values.report | jq('.ignored | length') | first }}"
      ignored_detail: "{{ outputs.select.values.report | jq('[.ignored[].key] |
        join(\"; \")') | first }}"
      off_total: "{{ outputs.select.values.report | jq('(.targets | length) +
        (.blocked | length) + (.ignored | length)') | first }}"

  - id: summary
    type: io.kestra.plugin.core.log.Log
    message: >-
      Trigger check on {{ inputs.kestra_url }} namespace {{ inputs.namespace }}:
      {{ outputs.counts.values.off_total }} trigger(s) switched off. {{
      outputs.counts.values.target_count }} on the desired_live allowlist and
      fixable from the API [{{ outputs.counts.values.target_detail }}]. {{
      outputs.counts.values.blocked_count }} allowlisted but disabled in the
      flow source, which needs a source change [{{
      outputs.counts.values.blocked_detail }}]. {{
      outputs.counts.values.ignored_count }} off and not on the allowlist, left
      alone [{{ outputs.counts.values.ignored_detail }}].

  - id: route
    type: io.kestra.plugin.core.flow.If
    condition: "{{ (outputs.counts.values.off_total | number) > 0 }}"
    then:
      - id: alert
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {{ {'text': 'Kestra trigger guard on ' ~ inputs.kestra_url ~ ' namespace ' ~ inputs.namespace ~ ': ' ~ outputs.counts.values.target_count ~ ' allowlisted trigger(s) are switched off and should be live (' ~ outputs.counts.values.target_detail ~ '). ' ~ outputs.counts.values.blocked_count ~ ' allowlisted trigger(s) are disabled in the flow source and need a code change (' ~ outputs.counts.values.blocked_detail ~ '). ' ~ outputs.counts.values.ignored_count ~ ' off trigger(s) are not on the allowlist and were left alone (' ~ outputs.counts.values.ignored_detail ~ ').' ~ (inputs.auto_remediate and not inputs.dry_run ? ' Switching the allowlisted ones back on behind approval.' : ' No change applied (auto_remediate=' ~ inputs.auto_remediate ~ ' dry_run=' ~ inputs.dry_run ~ ').')} | toJson }}

      - id: remediate_gate
        type: io.kestra.plugin.core.flow.If
        condition: "{{ inputs.auto_remediate and not inputs.dry_run }}"
        then:
          - id: act_gate
            type: io.kestra.plugin.core.flow.If
            condition: "{{ (outputs.counts.values.target_count | number) > 0 }}"
            then:
              - id: ceiling
                type: io.kestra.plugin.core.flow.If
                condition: "{{ (outputs.counts.values.target_count | number) >
                  inputs.max_enables }}"
                then:
                  - id: too_many
                    type: io.kestra.plugin.core.execution.Fail
                    errorMessage: "Refusing to switch on {{ outputs.counts.values.target_count }}
                      trigger(s), over the max_enables ceiling of {{
                      inputs.max_enables }}. Something switched off a whole
                      namespace. Look at why before raising the ceiling."
                else:
                  - id: approval
                    type: io.kestra.plugin.core.flow.Pause
                    description: Hold for a human to approve switching these triggers back on. No
                      answer within the window changes nothing.
                    pauseDuration: PT1H
                    behavior: RESUME
                    onResume:
                      - id: approved
                        type: BOOL
                        required: false
                        description: Set true to switch the listed triggers back on. Anything else
                          leaves them off.
                      - id: note
                        type: STRING
                        required: false
                        description: Optional reason, included in the confirmation log.

                  - id: approved_gate
                    type: io.kestra.plugin.core.flow.If
                    condition: "{{ outputs.approval.onResume.approved ?? false }}"
                    then:
                      - id: reenable
                        type: io.kestra.plugin.core.flow.Loop
                        description: One trigger at a time, so this remediation cannot fire concurrent
                          writes at the same trigger table.
                        concurrencyLimit: 1
                        values: "{{ outputs.counts.values.target_keys }}"
                        tasks:
                          - id: enable_one
                            type: io.kestra.plugin.kestra.triggers.Toggle
                            allowFailure: true
                            kestraUrl: "{{ inputs.kestra_url }}"
                            namespace: "{{ inputs.namespace }}"
                            flowId: "{{ fromJson(item.value).f }}"
                            trigger: "{{ fromJson(item.value).t }}"
                            enabled: true
                            auth:
                              username: "{{ inputs.kestra_user }}"
                              password: "{{ inputs.kestra_password }}"

                      - id: settle
                        type: io.kestra.plugin.core.flow.Sleep
                        duration: PT10S

                      - id: reverify
                        type: io.kestra.plugin.core.http.Request
                        description: Read the trigger state again and keep the targeted triggers that
                          are still switched off.
                        uri: "{{ inputs.kestra_url }}/api/v1/main/triggers/search"
                        method: GET
                        contentType: application/json
                        options:
                          auth:
                            type: BASIC
                            username: "{{ inputs.kestra_user }}"
                            password: "{{ inputs.kestra_password }}"
                        params:
                          filters[namespace][EQUALS]: "{{ inputs.namespace }}"
                          size: "500"

                      - id: verify_counts
                        type: io.kestra.plugin.core.output.OutputValues
                        values:
                          still_off_keys: "{{ outputs.reverify.body |
                            jq(outputs.counts.values.target_key_list ~ ' as $t |
                            [.results[] | select((.state.disabled == true) or
                            (.state.sourceDisabled == true)) | (.state.flowId +
                            \"/\" + .state.triggerId)] as $off | ($t - ($t -
                            $off))') | first | toJson }}"
                          still_off_count: "{{ outputs.reverify.body |
                            jq(outputs.counts.values.target_key_list ~ ' as $t |
                            [.results[] | select((.state.disabled == true) or
                            (.state.sourceDisabled == true)) | (.state.flowId +
                            \"/\" + .state.triggerId)] as $off | (($t - ($t -
                            $off)) | length)') | first }}"
                          next_runs: "{{ outputs.reverify.body | jq(outputs.counts.values.target_key_list
                            ~ ' as $t | [.results[] | select(((([.state.flowId +
                            \"/\" + .state.triggerId]) - $t) | length) == 0) |
                            .state.flowId + \"/\" + .state.triggerId + \" next
                            \" + (.state.nextEvaluationDate // \"never\")] |
                            join(\"; \")') | first }}"

                      - id: verify_gate
                        type: io.kestra.plugin.core.flow.If
                        condition: "{{ (outputs.verify_counts.values.still_off_count | number) > 0 }}"
                        then:
                          - id: enable_failed
                            type: io.kestra.plugin.core.execution.Fail
                            errorMessage: "Switching triggers back on did not land: {{
                              outputs.verify_counts.values.still_off_count }}
                              targeted trigger(s) are still off ({{
                              outputs.verify_counts.values.still_off_keys }}).
                              The API user may lack permission. The trigger may
                              also be disabled in the flow source, which no API
                              call can clear."
                        else:
                          - id: reenabled
                            type: io.kestra.plugin.core.log.Log
                            message: "Switched {{ outputs.counts.values.target_count }} trigger(s) back on
                              and confirmed it against the trigger state: {{
                              outputs.verify_counts.values.next_runs }}. Work
                              resumes at those times."
                    else:
                      - id: declined
                        type: io.kestra.plugin.core.log.Log
                        message: "Approval not granted, so nothing was switched on. {{
                          outputs.counts.values.target_count }} allowlisted
                          trigger(s) are still off: {{
                          outputs.counts.values.target_detail }}."
            else:
              - id: nothing_fixable
                type: io.kestra.plugin.core.log.Log
                message: "Nothing to switch on from the API. {{
                  outputs.counts.values.blocked_count }} allowlisted trigger(s)
                  are disabled in the flow source ({{
                  outputs.counts.values.blocked_detail }}) and {{
                  outputs.counts.values.ignored_count }} off trigger(s) are not
                  on the allowlist ({{ outputs.counts.values.ignored_detail
                  }})."
    else:
      - id: clean
        type: io.kestra.plugin.core.log.Log
        message: "Every trigger in namespace {{ inputs.namespace }} on {{
          inputs.kestra_url }} is enabled. Nothing is silently switched off."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {'text': 'Kestra disabled trigger reenable guard failed on ' ~ inputs.kestra_url ~ ' (execution ' ~ execution.id ~ '). The target may be unreachable. The API user may be wrong. A trigger it switched on may still be off. Check the logs.'} | toJson }}

triggers:
  - id: daily
    type: io.kestra.plugin.core.trigger.Schedule
    description: Shipped disabled and always in safe mode.
    cron: "20 7 * * *"
    disabled: true
    inputs:
      auto_remediate: false
      dry_run: true

A schedule gets switched off for a good reason. It pages too much during an incident. Or a new guard ships off while someone watches it. Then nobody turns it back on. The schedule never fires again and nothing complains, because the thing that would have complained is the thing that is off. No error, no alert, no execution. Every dashboard looks calm because nothing is running to report. The failure is the absence of work, which is the hardest kind to notice.

This flow reads the trigger state of whichever Kestra you point kestra_url at, finds the triggers that are off, then switches back on only the ones an operator has written down as desired_live. Then it reads the state again and fails the run if a trigger it targeted is still off.

This blueprint was created by zkasuran.

Safety

Restarting a schedule somebody stopped on purpose is worse than leaving it off, so the allowlist is the whole safety model, not a convenience.

It acts ONLY on triggers listed in desired_live as flowId/triggerId. A trigger that is off and not on the list is reported in the log plus the Slack message, then left exactly as it is. With the default empty list the flow changes nothing at all.
It looks at one namespace.
A change runs only when all of these hold: auto_remediate is true, dry_run is false, the count is at or under max_enables, a human approves the Pause. Any missing step changes nothing.
After the change it reads the trigger state again and fails the run if any targeted trigger is still off.

Findings

Two things turned up while running this against a real Kestra 2.0.5. Both shape the flow.

A trigger can be off in two different ways and only one of them is fixable from the API. GET /api/v1/main/triggers/search returns state.disabled, the stored flag an operator flips in the UI, next to state.sourceDisabled, which mirrors disabled: true in the flow source. io.kestra.plugin.kestra.triggers.Toggle writes the stored flag only. A source-disabled trigger stays off whatever the API does, because the fix is a change to the flow file. So the flow sorts the two apart, switches on the stored-flag ones, then reports the source-disabled ones as needing a code change instead of pretending to fix them.
Toggle logs 0 triggers found to toggle on a write that landed. It reads a count field the 2.0.5 API does not send, so the number is always zero. The re-read is the only trustworthy signal, which is exactly why this flow fails on the re-read rather than on the task log.

How it works

detect (io.kestra.plugin.core.http.Request, read-only) calls GET /api/v1/main/triggers/search with filters[namespace][EQUALS] set to the namespace. The modern filters[...] form is used on purpose, because the flat namespace= form is ignored on 2.0.5 and returns every trigger on the instance.
select sorts the switched-off triggers into three sets: targets (on the allowlist, stored flag off, fixable here), blocked (on the allowlist, off in the flow source) plus ignored (off and not on the allowlist).
counts and summary report all three sets with the last time each trigger fired.
route (io.kestra.plugin.core.flow.If):
- nothing off: clean logs it.
- something off: alert posts all three sets to Slack.
  - remediate_gate runs only when auto_remediate is true and dry_run is false.
  - act_gate skips straight to nothing_fixable when every off trigger is either source disabled or not allowlisted.
  - ceiling fails the run rather than acting when more than max_enables are off.
  - approval (io.kestra.plugin.core.flow.Pause) waits for a human. No answer changes nothing.
  - reenable (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) switches each one on with io.kestra.plugin.kestra.triggers.Toggle and enabled: true.
  - reverify reads the trigger state again. verify_gate fails the run if any targeted trigger is still off. If they are all live it logs the next scheduled run of each one.
errors posts to Slack when the target cannot be reached or the verify fails.
daily (io.kestra.plugin.core.trigger.Schedule), shipped disabled and in safe mode.

What you get

The triggers in a namespace that are switched off, each with the last time it fired, so a schedule that has not run in weeks is obvious.
The ones you declared should be live, switched back on, scoped by an allowlist, approved by a human and capped.
The ones that are off in the flow source, named separately, because those need a pull request rather than an API call.
A second read of the trigger state as proof, with the next scheduled run of each trigger, so the result is work resuming rather than a flag that moved.

Who it's for

Platform teams who disable a noisy schedule during an incident and need something that remembers to turn it back on.
Anyone who has found a nightly job that quietly stopped running months ago.
Teams who want the remediation step a trigger monitor stops short of, without it restarting something that was paused deliberately.

Why orchestrate this with Kestra

The guard is itself a Kestra flow, so watching Kestra needs no extra infrastructure. It runs on a schedule, posts to Slack, holds the change behind an approval, acts on one trigger at a time, then keeps the whole decision in the execution history. Point kestra_url at any instance to check a remote cluster from one place.

Tested end to end

On Kestra 2.0.5 OSS. The flow ran on one Kestra and pointed kestra_url at a second throwaway Kestra, so no real trigger was ever touched. A local HTTP sink stood in for Slack. The target held five flows with Schedule triggers in one namespace: two switched off through the API and on the allowlist, one switched off and deliberately NOT on the allowlist, one left enabled and on the allowlist, one carrying disabled: true in its own source. Before and after values come from the target's own triggers/search API, not from the flow log.

Run	Inputs	Result
Clean	allowlist of one enabled trigger	0 off, clean branch, run SUCCESS
Detect and report	3 off, `dry_run: true`	2 targets, 1 source blocked, 1 not allowlisted, Slack alert sent, all 3 still off afterwards
Remediate	`auto_remediate: true`, `dry_run: false`	Pause approved, 2 switched on, re-read found 0 still off, the target API confirmed both stored flags cleared. The non-allowlisted trigger stayed off
Work resumed	same run, one target on a one minute cron	the re-enabled trigger fired a real execution within the minute, proven by a new execution on the target plus a moving `lastTriggeredDate`
Nothing fixable	allowlist of only the source-disabled trigger	reported as needing a flow change, no Toggle issued, run SUCCESS
Declined	Pause resumed without approval	nothing switched on, both targets still off
Ceiling	`max_enables: 1` with 2 off	run failed before any change, both left off
Re-verify gate	a copy that also targets the source-disabled trigger	Toggle could not clear it, the re-read caught it, the run failed instead of reporting success
Target down	wrong port	detect failed, the errors block posted to Slack

Prerequisites

A Kestra API user that can read and update triggers in the namespace.
kestra_url reachable from the Kestra worker. The default http://localhost:8080 works when the flow checks the cluster it runs on.

Secrets

KESTRA_PASSWORD: the API password, used as the default for the kestra_password input.
SLACK_WEBHOOK_URL: used by alert and the errors block.
In Kestra OSS, provide each one as a base64-encoded environment variable prefixed SECRET_.

Inputs

Input	Default	Purpose
`kestra_url`	`http://localhost:8080`	Target Kestra API
`namespace`	`company.team`	The one namespace considered
`desired_live`	`[]`	Allowlist of `flowId/triggerId` that must be live
`max_enables`	`10`	Refuse to switch on more than this in one run
`kestra_user`	`admin@kestra.io`	API username
`kestra_password`	`{{ secret('KESTRA_PASSWORD') }}`	API password
`auto_remediate`	`false`	Allow the change
`dry_run`	`true`	Must be `false` for anything to change

Quick start

Add the KESTRA_PASSWORD and SLACK_WEBHOOK_URL secrets.
Run it once with the defaults to list every trigger in the namespace that is off.
Take the ones that should be live from that list and put them in desired_live as flowId/triggerId.
Run with auto_remediate: true and dry_run: false, then approve the Pause.
Enable the daily trigger, which always runs in safe mode.

How to extend

Open a ticket per source-disabled trigger, since those need a change to the flow file.
Keep desired_live in Git beside the flows, so the list of schedules that must be live is reviewed like code.
Alert on how long a trigger has been off, read from lastTriggeredDate, so a schedule off for a week pages louder than one off for an hour.
Loop over several kestra_url values to check more than one cluster.

Re-enable Kestra triggers that were never switched back on

Safety

Findings

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Re-enable Kestra triggers that were never switched back on

Safety

Findings

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Nightly Kestra flows backup to S3 with a SHA-256 drift manifest per namespace

Release Kestra Flows from Git with Smoke Tests and Automatic Rollback

Recover Kestra executions abandoned in PAUSED

Kestra instance retention policy with dry run, approval and audit CSV

Kestra Schedule Trigger Health Sentinel

Reap Kestra executions stuck in RUNNING

New to Kestra?