List icon
OutputValues icon
Request icon
Query icon
If icon
Fail icon
SlackIncomingWebhook icon
Pause icon
Toggle icon
Log icon
Schedule icon

Quarantine a repeatedly failing Kestra polling trigger

Quarantine a repeatedly failing Kestra polling trigger behind approval, verify scheduler state and keep healthy triggers running.

Categories
CoreInfrastructure

Safety

Disabling a polling trigger stops new work from that trigger.

Guard What it stops
Exact identifiers and enabled POLLING state Writes to neighbouring or source-disabled triggers
Both evidence thresholds and failOnTriggerError Selection based only on failed flow tasks
Safe defaults and an explicit approval Unattended quarantine
Full detector, source and scheduler re-read after approval Writes based on stale counts or a recovered, edited, deleted or disabled target
Scheduler read after Toggle Reporting an ineffective write as successful

How it works

  1. initial uses native flows.List to snapshot one target flow; before reads its polling-trigger scheduler state through the API because the plugin has no task for that state read.
  2. logs selects WARN/ERROR evaluation rows for the exact namespace, flow and trigger over time_range; native logs.Fetch can select the trigger pseudo-execution but cannot bound a read by duration.
  3. Native executions.Query reads FAILED trigger-created executions; counts selects the exact trigger and CREATED-to-FAILED state history, excluding executions that reached RUNNING.
  4. capacity refuses a read reaching the 500-row ceiling; candidate requires both positive thresholds and an enabled polling trigger with failOnTriggerError: true.
  5. With auto_remediate: true and dry_run: false, approval shows the first error and both counts; after approval, fresh_logs, fresh_failures and fresh_counts reapply both thresholds; prewrite and prewrite_state require unchanged source, enabled state and a newer completed evaluation matching the latest worker error.
  6. Native triggers.Toggle disables that exact trigger; verify reads scheduler state and fails unless disabled: true is present, then Slack records the trigger and approval note.

This blueprint was created by zkasuran.

What you get

  • Evaluation error-row and creation-time failure counts for one polling trigger.
  • A report-only default, approval record and verified scheduler quarantine.
  • Slack messages for candidates, completed quarantines and failed guard executions.

Who it's for

Platform teams responsible for Kestra polling triggers and noisy dependency failures.

Why orchestrate this with Kestra

Native execution history, Pause approval and the Toggle task keep detection and the decision in one execution. Both API credential inputs use SECRET to protect operator-supplied values in execution records.

Tested end to end

Kestra OSS 2.0.6 with an unreachable PostgreSQL polling trigger, a healthy SQLite control and ordinary flow-task failures. Independent target API reads found 60 WARN/ERROR evaluation rows and 15 failed evaluations before quarantine; no new rows or failed evaluations appeared over 247 seconds after disabling the one-minute trigger.

Case Execution Result
Credential protection 3peTscUn95IvtSZd8msSyB Raw inputs, outputs and logs checked
Healthy dependency 5YNvvSjEQDQDPbaxwtmqlV No candidate
Degraded dry run 12IMwmiHTzfJn0vBqpKS2M Candidate; no write
Declined approval 7fqF6t5lZJqp9i4IclU0iv No write
Approved quarantine RT8DFKm1nU3Nxp67Gj2LW Scheduler disabled=true
Repeat 2gBBoEhMkQHGa1xWUhkxfW No further Toggle
Source changed during approval 1hm1GBJZUNndL9B5W4lIJ4 Failed before Toggle
Proxy acknowledges a no-op 3oVkFVPtGBNAH7nQPza8tM Verification failed
Dependency recovered during approval 6p6iTBWgIydpOw2Xd3umVM Failed before Toggle
Real Toggle, verification unavailable 3hQnhuhzRPFC3VXVXE0a49 Failed; retry preserves quarantine

Additional cases cover both threshold boundaries, source-disabled targets, neighbouring namespaces and triggers, empty and saturated results, invalid identifiers, a concurrent second execution, deletion during approval, refused writes and slow or unavailable state reads. Healthy control, quarantine, approval-time source change and repeat were reproduced against a second fresh Kestra 2.0.6 target.

Prerequisites

  • A reachable target Kestra OSS 2.0.6 API and a polling trigger configured with failOnTriggerError: true.
  • A target API identity permitted to list flows, search logs and executions, read trigger state and toggle the selected trigger.

Secrets

  • KESTRA_USER and KESTRA_PASSWORD: target API identity.
  • SLACK_WEBHOOK_URL: Slack incoming webhook.

Inputs

Input Default Purpose
kestra_url, tenant_id http://localhost:8080, main Target API
namespace, flow_id, trigger_id company.team, target-flow, poll_dependency Exact target
time_range PT15M Historical duration window
min_log_errors, min_failed_executions 3, 3 Both thresholds must be met; allowed range 1 to 500
kestra_user, kestra_password Username and password secrets Basic authentication
auto_remediate, dry_run false, true Report-only defaults

Quick start

  1. Provide the API and Slack secrets plus the exact target coordinates.
  2. Run with the report-only defaults and review the candidate error and counts.
  3. For a confirmed incident, set auto_remediate: true and dry_run: false, wait for another failed polling evaluation during the Pause, then approve.
  4. Inspect the verified scheduler state and Slack record; fix the dependency before re-enabling the trigger.

How to extend

Raise both thresholds for a noisier environment while keeping exact target selection and approval.

Limits: Counts cover a historical window. After approval, a newer completed scheduler evaluation must match the latest structured worker evaluation-error date; this check depends on Kestra 2.0.6 log formatting and refuses missing evidence or approval before another evaluation. Three failed creation-time executions are the default repeated-event threshold; log rows only corroborate because one evaluation emits several rows. Lowering the execution threshold to one permits a transient incident. Other creation failures with the same CREATED-to-FAILED history can affect report counts; errors outside the window and failOnTriggerError=false are excluded. Toggle changes scheduler state and leaves YAML unchanged. This guard serializes its own executions only. Operators must have exclusive control of the chosen trigger during approval and verification: unrelated writers can race the final read because Toggle has no conditional-version property. Do not enable remediation where that ownership cannot be guaranteed. Fix the dependency before manually re-enabling the trigger.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.