AWS Backup Failed Job Sentinel

AWS Backup unifies backup scheduling across EC2, EBS, RDS, EFS, DynamoDB, and S3, but a failed job - an IAM permission that was revoked, a resource that was deleted mid-backup, a vault lock conflict - produces no proactive notification on its own; someone has to open the AWS Backup console and check. This blueprint lists recent jobs filtered server-side to FAILED and alerts Slack with the exact resource and reason the moment any exist.

How it works

1. periodic_backup_check (io.kestra.plugin.core.trigger.Schedule) runs every 6 hours, explicitly passing aws_region and lookback_hours through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
2. probe_failed_backup_jobs (io.kestra.plugin.aws.cli.AwsCLI) runs aws backup list-backup-jobs --by-state FAILED --by-created-after <lookback_hours ago>, capturing the result's BackupJobs list (and NextToken) as named task outputs via the plugin's own documented output-echo convention.
3. evaluate_backup_jobs (io.kestra.plugin.core.flow.If) branches on whether that list has any entries at all.
   • then (FAILED_JOBS_DETECTED): alert_failed_backup_jobs reports every failed job's resource type, ARN, and StatusMessage to Slack.
   • else (HEALTHY): log_all_backups_healthy records a clean sweep.
4. log_audit_complete always runs last, printing the failed job count regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - AWS Backup not being used in the region, or bad credentials, must not read as HEALTHY.

What you get

• A HEALTHY/FAILED_JOBS_DETECTED answer computed by AWS Backup's own server-side state and recency filters, not by paging through every job (completed and not) and filtering client-side.
• The exact resource type, ARN, and StatusMessage for every failed job in the Slack alert itself, instead of requiring someone to open the AWS Backup console to see what failed and why.
• One governance check across every resource type AWS Backup supports (EC2, EBS, RDS, EFS, DynamoDB, S3, Aurora), instead of a separate backup-monitoring script per service.
• A worked example of io.kestra.plugin.aws.cli.AwsCLI driving a read-only audit via a plain aws backup CLI call, reusing the same corrected output-capture convention (newlines only, not spaces, stripped before the marker) as this repository's CloudFormation and AWS Config sentinel blueprints.

Who it's for

• Platform and infrastructure teams relying on AWS Backup across multiple resource types who need a single, immediate signal when any backup job fails.
• On-call engineers who want one Slack alert with the exact resource and failure reason instead of opening the AWS Backup console to check manually.
• Anyone evaluating the AWS plugin who wants a worked example of AwsCLI used for a read-only, time-windowed governance audit rather than a provisioning command.

Why orchestrate this with Kestra

AWS Backup itself has no cross-resource, cross-vault "notify me the moment any job fails" primitive beyond what you wire up separately in EventBridge and SNS per vault - configuration that then has to be kept in sync as vaults and plans change. Kestra supplies the schedule, the HEALTHY/FAILED_JOBS_DETECTED classification as a first-class branch, and an execution history that shows exactly when a backup started failing and which resource was affected.

Prerequisites

• An AWS account using AWS Backup with at least one backup plan active in the target region (update aws_region).
• AWS credentials with backup:ListBackupJobs permission - entirely read-only, cannot modify any backup job, vault, or plan.
• A Slack incoming webhook for alerts.

This blueprint has no local, account-free testing path. ListBackupJobs is not implemented in LocalStack at all (verified from LocalStack's own documented API coverage for the Backup service), and LocalStack now additionally requires its own account and auth token as of March 2026 regardless. A real AWS account with AWS Backup actually in use is the only way to exercise this blueprint meaningfully.

Secrets

• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY: credentials for io.kestra.plugin.aws.cli.AwsCLI.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_failed_backup_jobs and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_AWS_SECRET_ACCESS_KEY=$(echo -n '<your-secret-key>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

• aws_region (STRING, default us-east-1): region whose backup jobs are audited.
• lookback_hours (INT, default 24): recency window passed to --by-created-after.

Outputs

• outputs.probe_failed_backup_jobs.vars.BackupJobs: the list of failed jobs, each with ResourceArn, ResourceType, ResourceName, BackupVaultName, StatusMessage, CreationDate, and more (verified fields from AWS Backup's own ListBackupJobs API reference).
• outputs.probe_failed_backup_jobs.vars.NextToken: pagination token, present when more failed jobs exist than fit in one page - not consumed by this blueprint (see Pitfalls).

Quick start

1. Set up an AWS Backup plan in a real AWS account covering at least one resource (for example, a small EBS volume).
2. Add the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually after a successful backup and confirm log_all_backups_healthy fires.
4. Force a failure (for example, revoke the backup IAM role's permission to the target resource temporarily, then trigger an on-demand backup) and re-run within lookback_hours to confirm alert_failed_backup_jobs fires with the real resource ARN and status message.
5. Enable periodic_backup_check once you trust the check against your real account.

How to extend

• Loop over several aws_region values with io.kestra.plugin.core.flow.Loop to audit every region where AWS Backup is in use in one run.
• Add a second probe using --by-resource-type to break the Slack alert into separate, resource-type-specific messages when failures span multiple services.
• Handle NextToken with a follow-up AwsCLI call using --starting-token if an account has enough failed jobs to paginate - this blueprint's single call only sees the first page.

Pitfalls

• Pagination is not handled. If more failed jobs exist than fit in one list-backup-jobs page, NextToken will be present in the output but this blueprint does not follow it - see How to extend.
• lookback_hours must be wide enough to cover the gap between scheduled runs, or a failure that happened between two checks and was already resolved before the next one could still surface (which is arguably correct - a failure that happened is worth knowing about even if since resolved) or, conversely, a failure just outside the window could be missed entirely if lookback_hours is set shorter than the check interval.
• The output-capture convention (tr -d '\n' | xargs -0 -I {} echo '::{"outputs":{}}::') deliberately does not strip spaces, the same corrected pattern used in this repository's CloudFormation and AWS Config sentinel blueprints, since StatusMessage is a human-readable sentence that stripping spaces would corrupt.
• ListBackupJobs is not implemented in LocalStack at all, on top of LocalStack now requiring its own account since March 2026 - neither issue is specific to this blueprint; both are disclosed here so they are not mistaken for a bug in it.
• This blueprint is UNTESTED against a live AWS account. The CLI command, its --by-state and --by-created-after filters, and the BackupJobs / StatusMessage / ResourceArn output fields are taken verbatim from AWS's own CLI reference documentation - but no Kestra engine was run to execute this flow end to end.

Links

• Kestra AWS plugin documentation
• AWS Backup ListBackupJobs API reference
• AWS CLI list-backup-jobs reference
• Kestra secrets (OSS vs Enterprise)