AWS Step Functions Execution Failure Rate Sentinel

A Step Functions state machine that starts failing every execution - a broken downstream integration, an expired credential inside a Task state, a Lambda that started throwing - has no first-class "alert me" primitive of its own; CloudWatch alarms have to be configured and wired separately for every state machine. This blueprint reads the same ExecutionsFailed metric CloudWatch alarms would use, classifies it, and alerts Slack directly from a Kestra flow instead.

How it works

1. periodic_failure_check (io.kestra.plugin.core.trigger.Schedule) runs every 15 minutes, explicitly passing aws_region, state_machine_arn, query_window_minutes, and max_failed_executions through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
2. probe_failed_executions (io.kestra.plugin.aws.cloudwatch.Query) queries the AWS/States namespace's ExecutionsFailed metric, scoped to state_machine_arn via the StateMachineArn dimension, with periodSeconds set to the full lookback window so CloudWatch returns a single aggregated Sum datapoint rather than one per minute.
3. evaluate_metric_presence (io.kestra.plugin.core.flow.If) branches on whether any datapoint came back at all (count).
   • then (NO_DATA): log_no_metric_data reports that no executions ran in this window at all.
   • else: nested evaluate_failure_threshold (io.kestra.plugin.core.flow.If) branches on whether the aggregated Sum exceeds max_failed_executions.
     • then (FAILURES_DETECTED): alert_failures_detected reports the count to Slack.
     • else (HEALTHY): log_executions_healthy records a clean check.
4. log_audit_complete always runs last, printing the datapoint count regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable CloudWatch API or bad credentials must not read as any of the three handled states.

What you get

• A NO_DATA/HEALTHY/FAILURES_DETECTED answer sourced from the exact same metric CloudWatch's own native alarms would use, with zero risk to the state machine itself - no execution is ever started, stopped, or redriven by this sentinel.
• A distinct NO_DATA state instead of silently treating a quiet window as either healthy or alarming, which matters for infrequently invoked state machines.
• A worked example of io.kestra.plugin.aws.cloudwatch.Query with periodSeconds deliberately set equal to the full lookback window, so the flow reads one pre-aggregated Sum value instead of summing several datapoints itself in Pebble.
• A required AWS IAM permission set (cloudwatch:GetMetricStatistics only) that cannot touch Step Functions executions at all, by construction.

Who it's for

• Platform teams running Step Functions state machines who need an early signal when a workflow starts failing, without wiring a separate CloudWatch alarm and SNS topic per state machine.
• On-call engineers who want one Slack alert with the exact failure count instead of opening the CloudWatch or Step Functions console to check.
• Anyone evaluating the AWS CloudWatch plugin who wants a worked example of Query scoped to a specific resource via a dimension, with the statistic deliberately chosen to match AWS's own documented guidance for count-type metrics.

Why orchestrate this with Kestra

CloudWatch alarms can notify on a threshold, but configuring one per state machine, wiring it to SNS, and keeping that configuration in sync with the state machines that actually exist is its own maintenance burden, separate from the flow that needs to react. Kestra supplies the schedule, the NO_DATA/HEALTHY/FAILURES_DETECTED classification as a first-class branch, and an execution history that shows exactly when a state machine started failing and how many executions were affected.

Prerequisites

• An AWS account with a deployed Step Functions state machine that has run at least once (update aws_region, state_machine_arn).
• AWS credentials with cloudwatch:GetMetricStatistics permission - nothing from Step Functions itself is required, since this blueprint never calls the Step Functions API.
• A Slack incoming webhook for alerts.

This blueprint has no local, account-free testing path. LocalStack's Community image now requires its own account and auth token as of March 2026 (it is no longer a plain, no-signup Docker image), and even where available, LocalStack's CloudWatch metrics support is limited. A real AWS account is the practical prerequisite here, the same baseline every other AWS-targeting blueprint in this repository assumes.

Secrets

• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY: credentials for io.kestra.plugin.aws.cloudwatch.Query.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_failures_detected and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_SLACK_WEBHOOK_URL=$(echo -n '<your-webhook-url>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

• aws_region (STRING, default us-east-1).
• state_machine_arn (STRING, default <your-state-machine-arn>, must be replaced).
• query_window_minutes (INT, default 15): drives both window and periodSeconds on the probe task.
• max_failed_executions (INT, default 0): failure-count ceiling before FAILURES_DETECTED fires.

Outputs

• outputs.probe_failed_executions.count: number of datapoints returned (expected 0 or 1, since periodSeconds equals the full window).
• outputs.probe_failed_executions.series[0].sum: the aggregated failed-execution count for the window, only present when count is 1 (supports sum or Sum).

Quick start

1. Deploy a Step Functions state machine in a real AWS account and let it run at least once successfully.
2. Add the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually against a healthy state machine and confirm log_executions_healthy or log_no_metric_data fires, depending on recent execution volume.
4. Force an execution to fail (for example, by starting one with input that a Task state's Lambda rejects) and re-run within query_window_minutes to confirm alert_failures_detected fires.
5. Enable periodic_failure_check once you trust the check against your real state machine.

How to extend

• Loop over several state_machine_arn values with io.kestra.plugin.core.flow.Loop to audit every critical state machine in an account in one run.
• Add a second io.kestra.plugin.aws.cloudwatch.Query probe on ExecutionsTimedOut (also documented under the same AWS/States namespace) to catch stuck-and-timing-out executions as a distinct state from outright failures.
• Add a third probe on the account-level OpenExecutionCount metric (dimensionless, no StateMachineArn needed) to additionally catch a state machine approaching its open-execution quota before any individual execution fails.

Pitfalls

• CloudWatch metrics are delivered on a best-effort basis, per AWS's own Step Functions monitoring documentation - a data point can arrive late, or occasionally not at all, so NO_DATA is not a perfectly reliable signal that zero executions actually failed, only that CloudWatch has not reported any yet.
• AWS's own documentation recommends the Sum statistic for count-type metrics like ExecutionsFailed, not the task's own default of Average - this blueprint sets statistic: "Sum" explicitly for exactly that reason; leaving it at the default would silently report an averaged value instead of a true count.
• periodSeconds must be kept in sync with window. Both are derived from the single query_window_minutes input specifically so they cannot drift apart; setting them independently risks CloudWatch returning multiple smaller datapoints that this flow's single series[0] read would then silently ignore all but the first of.
• This blueprint has no local, account-free testing path. LocalStack now requires its own account and token since March 2026, so a real AWS account is the practical prerequisite, verified and disclosed rather than glossed over.
• This blueprint is UNTESTED against a live AWS account. The ExecutionsFailed metric, the AWS/States namespace, the StateMachineArn dimension, and the Sum-statistic recommendation are all taken verbatim from AWS's own Step Functions CloudWatch monitoring documentation, and the io.kestra.plugin.aws.cloudwatch.Query properties and outputs are taken from the plugin's own source on main - but no Kestra engine was run to execute this flow end to end.

Links

• Kestra AWS plugin documentation
• AWS Step Functions CloudWatch metrics documentation
• Kestra secrets (OSS vs Enterprise)