OutputValues icon
If icon
SlackIncomingWebhook icon
Write icon
Log icon
FluxTrigger icon

InfluxDB Flux Threshold Breach Severity Sentinel

Fire on InfluxDB Flux threshold breaches, alert Slack on widespread breaches, auto-acknowledge isolated ones, and log the result for platform teams.

Categories
Infrastructure

A single sensor spiking past a threshold for one poll is noise; the same threshold breached across many rows in the same poll is an incident. InfluxDB's Flux query language can filter down to exactly the breaching rows, but it has no concept of "isolated blip versus widespread event" and no built-in way to page a channel only when the second case happens. This blueprint turns a FluxTrigger poll that already found at least one breaching row into that distinction, auto-containing the quiet case and paging Slack only for the loud one.

This blueprint was created by Mohit Jeswani.

How it works

  1. watch_threshold_breach (io.kestra.plugin.influxdb.FluxTrigger) polls a Flux query that already filters down to rows exceeding the watched threshold and starts an execution only when at least one row comes back, with fetchType: FETCH so the rows themselves are available as trigger.rows.
  2. evaluate (io.kestra.plugin.core.output.OutputValues) reads trigger.size, classifies the poll as ISOLATED_BREACH or WIDESPREAD_BREACH against widespread_breach_count, and builds a human-readable summary of the breaching rows with a jq filter.
  3. route (io.kestra.plugin.core.flow.If):
    • alert: posts the breach count and row summary to Slack.
    • ack_isolated_breach: writes a line-protocol acknowledgment point back to InfluxDB so a dashboard can tell the isolated breach was already seen, without paging anyone.
  4. summary logs the count and severity, closing the governance cycle.
  5. errors alerts Slack if the flow itself fails, for example an expired token.

What you get

  • A breach reported with the real breaching rows attached, not just "something crossed the threshold."
  • A severity split that only pages Slack for a widespread breach, so a single flapping sensor does not become alert fatigue.
  • A written, queryable acknowledgment trail for every isolated breach this flow contained on its own.

Who it's for

  • Platform and IoT/observability teams running InfluxDB who want a threshold breach to self-triage by scale before it reaches a human.
  • Teams evaluating the InfluxDB plugin who want a worked example of FluxTrigger used as a standing breach sentinel, distinct from this repo's existing InfluxDB blueprints, which audit bucket retention, series cardinality, and anomaly scores on a schedule rather than firing in response to a live threshold breach.

Why orchestrate this with Kestra

Flux can filter rows past a threshold in one query, but it has no scheduler, no severity policy, and no Slack integration of its own. Kestra keeps widespread_breach_count as a flow input, turns a FluxTrigger poll into a severity decision, and keeps a full execution history of every breach this flow has seen and how it was handled.

Prerequisites

  • An InfluxDB 2.x (or 3.x with the v2 compatibility API) instance with a token that can query and write to the target bucket.
  • A bucket with at least one measurement/field you can push past a threshold, to confirm the filter logic against.
  • A Slack incoming webhook URL.

A local InfluxDB instance can be run via the official influxdb:2.7 Docker image for a fully local end-to-end test, no managed account required.

Secrets

  • INFLUXDB_URL: the InfluxDB server URL used by both watch_threshold_breach and ack_isolated_breach. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_INFLUXDB_URL.
  • INFLUXDB_TOKEN: the InfluxDB authentication token. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_INFLUXDB_TOKEN.
  • SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input Default Purpose
widespread_breach_count 5 Breaching-row count per poll that escalates from isolated to widespread

Outputs

  • {{ trigger.rows }} / .size: the breaching rows for this poll and how many there are, as FluxTrigger reports them.
  • {{ outputs.evaluate.values.breach_count }} / .breach_summary: the row count and a semicolon-joined field=value at time summary of each breaching row.
  • {{ outputs.evaluate.values.severity }}: ISOLATED_BREACH or WIDESPREAD_BREACH.

Pitfalls

  • FluxTrigger declares its own bucket property, but its evaluate() method builds the internal FluxQuery from only connection, org, query, and fetchType, bucket is never passed through and has no effect on which data is queried. The bucket that actually matters is the one embedded in the Flux query string itself (the from(bucket: "...") call). This blueprint deliberately omits bucket from the trigger to avoid implying it does something it does not; verified by reading FluxTrigger.java's evaluate() directly against its own field declarations.
  • connection, org, and query are trigger-level configuration, not flow inputs; a polling trigger has no per-execution input overrides the way a Schedule trigger does, so pointing this at a different bucket or threshold means editing the trigger block directly.
  • trigger.rows keys mirror whatever Flux columns the query selects (typically _time, _field, _value, _measurement); a query that renames or drops those columns needs the breach_summary jq filter adjusted to match.
  • ack_isolated_breach writes to the same bucket the trigger watches; if the acknowledgment point's own fields happen to match the trigger's filter, a loose threshold could re-match its own acknowledgment on the next poll. The breach_ack measurement name is chosen to be distinct from the watched sensor measurement specifically to avoid that.

Quick start

  1. Add the INFLUXDB_URL, INFLUXDB_TOKEN, and SLACK_WEBHOOK_URL secrets.
  2. Run a local influxdb:2.7 container, create a bucket, and set watch_threshold_breach's query to match a real measurement/field/threshold in it.
  3. Enable the trigger, write one breaching point, and confirm ack_isolated_breach runs and summary logs ISOLATED_BREACH.
  4. Write widespread_breach_count or more breaching points in one poll window to confirm alert fires instead.

How to extend

  • Add a second If branch that escalates to a paging tool instead of Slack once breach_count crosses a second, higher threshold.
  • Chain a downstream task that reads ack_isolated_breach's acknowledgment points on a schedule to build a rolling "isolated breaches this week" report.
  • Scope the Flux query to a specific tag (host, device, region) per flow instance to run one sentinel per fleet segment instead of one global sentinel.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.