Webhook icon
Schedule icon
WorkingDirectory icon
Commands icon
Docker icon
Script icon
If icon
Create icon
SlackIncomingWebhook icon
Log icon

Audit Which Databases and Stores Are Missing From Backup Scope

Diff the asset registry against backup job targets, score coverage percentage, open a GitHub issue for uncovered stores, and alert Slack.

Categories
Infrastructure

How it works

  1. post_coverage_snapshot (io.kestra.plugin.core.trigger.Webhook, keyed) accepts the inventory POST from the backup reporting job. daily_coverage_sweep (io.kestra.plugin.core.trigger.Schedule) ships disabled: true as a fallback net once an operator switches it on.
  2. coverage_workspace (io.kestra.plugin.core.flow.WorkingDirectory) opens one shared directory so the writer and the differ read the same file, because default tasks are stateless and the temp directory is purged when a task completes.
  3. write_coverage_report (io.kestra.plugin.scripts.shell.Commands on the io.kestra.plugin.scripts.runner.docker.Docker runner, image python:3.12-slim) writes the posted body into posted_report.json with a quoted heredoc, leaving an empty document when the flow is executed from the UI.
  4. diff_coverage (io.kestra.plugin.scripts.python.Script on python:3.12-slim) prefers the posted report and falls back to the asset_registry and backup_targets inputs, then normalizes names, set-diffs the two sides, scores the coverage percentage, and collects the registry assets that no target protects alongside the targets that protect nothing in the registry.
  5. check_coverage (io.kestra.plugin.core.flow.If) branches on the verdict. Uncovered assets run file_coverage_issue (io.kestra.plugin.github.issues.Create), which opens a labelled issue naming every unprotected asset, and notify_uncovered, the Slack webhook carrying the coverage percentage. A report that could not be diffed runs check_diff_error, a second If whose alert_unreadable_report branch pages Slack, and only full coverage reaches log_full_coverage.
  6. The errors block alerts Slack whenever the flow itself breaks, because a dead auditor must never look like complete protection.

What you get

  • A coverage percentage for the whole registry, computed by set-diff instead of by trusting the backup tool's own success count.
  • A GitHub issue that lists every unprotected asset with its type and tier, so the gap has an owner and a thread.
  • A Slack alert with the exact asset names, delivered the moment the inventory is diffed.
  • Orphaned backup targets surfaced separately, since backups protecting nothing in the registry are wasted storage.
  • Gateable outputs, verdict, coverage_pct, uncovered_assets, asset_count, and orphan_targets for any parent audit flow.

Who it's for

  • Infrastructure and platform teams who own the backup estate and need to know what is outside it.
  • Data engineers responsible for gold-tier stores who want an audit trail instead of an assumption.
  • Compliance-minded operators who must show which registered assets were covered on a given day.

Why orchestrate this with Kestra

A set-diff is one loop, but an audit needs the state around it, a keyed webhook for the inventory job, a hermetic Python runtime pinned in the flow file, a tracked issue when coverage breaks, and an execution record proving the audit ran against that exact registry. Kestra supplies all of that declaratively, with GitHub and Slack routing plus an error handler that keeps a broken feed from reading as full coverage. A script on a cron box gives you none of that, it only protects the people who remember to read its output.

Prerequisites

  • A worker with Docker access, both the writer and the differ run in python:3.12-slim.
  • A Slack incoming webhook for coverage gap alerts, unreadable report pages, and failure alerts.
  • A GitHub token that can create issues on the tracking repository.
  • An inventory step that POSTs both lists, a registry array of assets and a backups array of protected targets.

Secrets

  • BACKUP_AUDIT_WEBHOOK_KEY: key segment guarding the webhook URL, anyone holding it can trigger an audit.
  • GITHUB_TOKEN: GitHub token used to open the coverage gap issue on the tracking repository.
  • SLACK_WEBHOOK_URL: Slack incoming webhook URL used for the gap alert, the unreadable report page, and failures.

Quick start

  1. Add the secrets above to your Kestra namespace and make sure the backup and coverage labels exist on the tracking repository.
  2. Run the flow once with the default inputs and confirm posted_report.json lands in the workspace and the outputs carry verdict, coverage_pct, and uncovered_assets.
  3. Add a sixth registry asset with no matching backup target, re-run, and watch verdict flip to uncovered with the new name in uncovered_assets and an issue opened on the repository.
  4. Point the inventory job at /api/v1/{tenant}/executions/webhook/{namespace}/backup-coverage-auditor/{BACKUP_AUDIT_WEBHOOK_KEY} once its POST body carries registry and backups.

How to extend

  • Audit several environments by wrapping this flow in a Subflow per environment and aggregating the coverage verdicts in a parent flow.
  • Enforce tiers by extending the diff to report gold-tier gaps separately, then escalate those ahead of bronze ones with an If on uncovered_details.
  • Drive a remediation ticket per uncovered asset with io.kestra.plugin.core.flow.Loop over uncovered_details.
  • Archive each audited snapshot to object storage with io.kestra.plugin.aws.s3.Upload so coverage history is queryable later.
  • Replace the inventory POST with a io.kestra.plugin.git.Clone of a registered-assets repository when the registry lives in Git.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.