Webhook icon
Schedule icon
WorkingDirectory icon
Clone icon
Commands icon
Docker icon
Script icon
Process icon
If icon
Create icon
SlackIncomingWebhook icon
Fail icon
Log icon

Prometheus Alert Rule Linting Gate with promtool

Lint Prometheus alert rules with promtool before merge. Escalate every violation as a GitHub issue, a Slack alert, and a failed Kestra execution.

Categories
Infrastructure

Merge gates usually test application code and leave monitoring configuration to luck, so a broken alert expression only surfaces when the pager fails to fire. This blueprint lints the rules themselves. A keyed webhook fires when your CI sees the rules directory change, the repository is checked out into a shared working directory, and promtool check rules runs inside the official Prometheus image with every lint check enabled, alongside promtool check config when a prometheus.yml sits at the repository root. A Python step turns the raw promtool output into a violations list with the file, the rule, and the message for each hit, and a single branch decides whether the run goes quiet or loud. On any hit the flow opens a labeled GitHub issue with the full list, pings Slack with the first few entries, and fails the execution so the merge stays blocked.

How it works

  1. The on_rules_change trigger (io.kestra.plugin.core.trigger.Webhook, guarded by the RULES_WEBHOOK_KEY secret) starts a run whenever your CI posts that files under the rules directory changed, and the nightly_lint io.kestra.plugin.core.trigger.Schedule trigger offers a daily cron that ships disabled: true.
  2. workspace (io.kestra.plugin.core.flow.WorkingDirectory) opens one shared filesystem for the checkout, the container, and the parser.
  3. clone_repository (io.kestra.plugin.git.Clone) checks out {{ inputs.repository }} at {{ inputs.branch }}, authenticating with GITHUB_TOKEN.
  4. run_promtool (io.kestra.plugin.scripts.shell.Commands on the io.kestra.plugin.scripts.runner.docker.Docker runner, image prom/prometheus:v2.53.0) finds every yaml and yml file under {{ inputs.rules_dir }}, writes the sorted list to rules-manifest.txt, runs promtool check rules --lint=all over that list, and appends promtool check config --lint=all prometheus.yml when the config file exists. Exit codes are absorbed with || true so the task stays green and the parser, not promtool, decides the verdict. Both files are declared as outputFiles.
  5. assess (io.kestra.plugin.scripts.python.Script on the io.kestra.plugin.core.runner.Process runner) pulls those two files in through inputFiles, walks the Checking and FAILED blocks, and emits verdict, gate_passed, files_checked, violation_count, the violations list, violations_markdown, and violations_brief through the outputs protocol.
  6. lint_gate (io.kestra.plugin.core.flow.If) opens the escalation path when violation_count is above zero: open_lint_issue (io.kestra.plugin.github.issues.Create) files the markdown list under three labels, alert_slack posts the count and the first hits, and fail_gate (io.kestra.plugin.core.execution.Fail) turns the execution red.
  7. A clean run falls through to log_clean (io.kestra.plugin.core.log.Log), while the errors handler pings Slack whenever the checkout, the container, or the parser itself breaks.

What you get

  • Alert rule syntax problems and lint findings caught in CI instead of during the next incident.
  • One violation entry per hit with the file, the rule name, and the message promtool printed.
  • A labeled GitHub issue that carries the whole list into the backlog engineers already work from.
  • A red execution that branch protections and downstream flow triggers can block on.
  • A quiet logged all-clear plus verdict, gate_passed, files_checked, violation_count, and a full lint_report for other flows.
  • A dated execution history that proves the rules directory was checked on every change.

Who it's for

  • SRE and platform teams who own a Prometheus rules repository and review alert changes in pull requests.
  • Observability leads who want duplicate rules, template errors, and malformed PromQL caught before merge.
  • Platform engineers wiring monitoring configuration into the same gates they already apply to application code.

Why orchestrate this with Kestra

A promtool step in CI tells you a file failed, but it does not open the issue, tell the team, or give the gate a state anyone can query later. Kestra supplies the keyed event trigger, a hermetic container so the promtool version lives in the flow rather than in a runner image someone maintains, a branch that separates escalation from the all-clear, and routing into GitHub and Slack from one declarative file. Because the check runs as an execution, every lint carries inputs, outputs, and logs you can replay, and the same flow can be called as a subflow from any deploy pipeline that must not proceed with broken rules.

Prerequisites

  • A worker with Docker access, since the lint runs inside the prom/prometheus:v2.53.0 image.
  • A task working directory the container can write to, because that image runs as nobody and the manifest and report files are written there.
  • A GitHub repository whose rule files live under a directory of yaml files and that allows issue creation with the provided token.
  • Python 3 on the Kestra worker for the parser, or switch assess to a Docker task runner.
  • A CI job that POSTs to the webhook whenever the rules directory changes.

Secrets

  • GITHUB_TOKEN: personal access token used both to clone the repository and to open the issue on violations.
  • RULES_WEBHOOK_KEY: key segment guarding the webhook URL, treat it like a password.
  • SLACK_WEBHOOK_URL: Slack incoming webhook used for violation and failure alerts.

Quick start

  1. Add the three secrets under Namespace, Secrets, then set repository and rules_dir to match your project layout.
  2. Run the flow once by hand and read verdict plus lint_report from the Outputs tab to confirm promtool is checking the files you expect.
  3. Point your CI at the webhook so every rules change starts a check:
# webhook target to paste into CI
https://your-kestra.example.com/api/v1/main/executions/webhook/company.team/prometheus-alert-rule-lint-gate/$RULES_WEBHOOK_KEY
  1. When you also want a nightly safety net, flip the schedule:
- id: nightly_lint
  type: io.kestra.plugin.core.trigger.Schedule
  cron: "0 5 * * *"
  disabled: false

How to extend

  • Narrow the findings by changing --lint=all to --lint=duplicate-rules, or add --lint-fatal if you want promtool to stop at the first problem.
  • Move promtool test rules into the same container command so unit test expectations gate alongside syntax.
  • Lint several rules directories by overriding rules_dir per call or by looping over a list of paths.
  • Replace open_lint_issue with a pull request comment or a ticket in Jira or Linear, or chain a remediation flow from the FAILED state.
  • Reuse the verdict as a subflow in every deploy pipeline with io.kestra.plugin.core.flow.Subflow.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.