Audit Azure DevOps Environment Approvals Against Their SLA

id: azure-devops-environment-approval-audit
namespace: company.team
description: |
  Release pipelines hang for days when Azure DevOps environment approvals
  stall past their SLA or get silently auto-bypassed, and nobody finds out
  until a stakeholder asks where the release went. A keyed webhook receives
  the pending approval snapshot from your audit job, a shared working
  directory stages a deterministic fixture for trial runs, and Python ages
  every pending approval against the configured SLA while counting
  auto-bypasses. Overdue or bypassed windows open a GitHub issue, post a
  Slack alert and fail the gate, while a clean window records an all-clear
  log, and the verdict, the overdue count, the oldest age and the worst
  approval are published as flow outputs.

triggers:
  - id: on_approval_snapshot
    type: io.kestra.plugin.core.trigger.Webhook
    description: Your Azure DevOps audit job or release status page POSTs the
      pending environment approval snapshot here so the SLA is scored the moment
      new approvals land.
    key: "{{ secret('AZURE_DEVOPS_APPROVALS_WEBHOOK_KEY') }}"
  - id: daily_sla_sweep
    type: io.kestra.plugin.core.trigger.Schedule
    description: Disabled daily 06:00 UTC net that rescores the staged snapshot,
      shipped off until the webhook path is trusted in your environment.
    cron: "0 6 * * *"
    disabled: true

inputs:
  - id: approval_sla_hours
    type: FLOAT
    defaults: 24.0
    description: Hours a single environment approval may sit pending before the
      audit counts it as overdue.
  - id: project_name
    type: STRING
    defaults: Fabrikam-Fiber
    description: Azure DevOps project the snapshot was taken from, echoed in the
      issue title, the issue body and the Slack alert.
  - id: github_repository
    type: STRING
    defaults: your-org/release-ops
    description: Repository in owner/repo form that receives the audit issue when
      the gate escalates.

tasks:
  # --- 1. approval workspace ---
  - id: approval_workspace
    type: io.kestra.plugin.core.flow.WorkingDirectory
    description: One shared directory so the staged approval snapshot and the SLA
      scorer read the same file for the whole run.
    tasks:
      # --- 2. snapshot fixture ---
      - id: write_approvals_fixture
        type: io.kestra.plugin.scripts.shell.Commands
        description: Write a deterministic approval snapshot fixture with a heredoc, one
          production approval pending 36.5 hours and one auto-bypassed approval,
          so the scorer always has a snapshot to read when no audit job has
          posted.
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
          entryPoint: []
        containerImage: alpine:3.20
        commands:
          - |
            set -u
            cat > approvals.json <<'JSON'
            {
              "project": "Fabrikam-Fiber",
              "collected_at": "2026-10-15T06:00:00Z",
              "approvals": [
                {"id": 901, "pipeline": "Fabrikam-Fiber-CI", "environment": "Production", "status": "pending", "age_hours": 36.5, "approver_group": "Release-Managers"},
                {"id": 902, "pipeline": "Fabrikam-Fiber-CI", "environment": "Staging", "status": "approved", "age_hours": 2.0, "approver_group": "Release-Managers"},
                {"id": 903, "pipeline": "Fabrikam-Fiber-Web", "environment": "Production", "status": "auto_bypassed", "age_hours": 8.0, "bypass_reason": "timeout policy"},
                {"id": 904, "pipeline": "Fabrikam-Fiber-DB", "environment": "Production", "status": "pending", "age_hours": 6.0, "approver_group": "DBA-On-Call"}
              ]
            }
            JSON
            echo "wrote $(wc -c < approvals.json) bytes of approval snapshot"

      # --- 3. SLA score ---
      - id: score_approvals
        type: io.kestra.plugin.scripts.python.Script
        description: Read the posted snapshot or the staged fixture, age every pending
          approval against the SLA hours, count auto-bypassed approvals, and
          emit the verdict, the counters and the oldest overdue approval through
          the outputs protocol.
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        containerImage: python:3.12-slim
        env:
          APPROVALS_JSON: "{{ trigger.body ?? '' }}"
          APPROVAL_SLA_HOURS: "{{ inputs.approval_sla_hours }}"
        script: |
          import json
          import os

          SLA_DEFAULT_HOURS = 24.0


          def as_float(value, fallback=0.0):
              try:
                  return float(value)
              except (TypeError, ValueError):
                  return fallback


          def load_snapshot():
              raw = (os.environ.get("APPROVALS_JSON") or "").strip()
              if raw:
                  try:
                      return json.loads(raw)
                  except ValueError:
                      return None
              path = "approvals.json"
              if not os.path.exists(path) or os.path.getsize(path) == 0:
                  return None
              try:
                  with open(path, "r", encoding="utf-8") as handle:
                      return json.load(handle)
              except ValueError:
                  return None


          sla_hours = as_float(os.environ.get("APPROVAL_SLA_HOURS"), SLA_DEFAULT_HOURS)

          outputs = {
              "parse_ok": False,
              "verdict": "UNKNOWN",
              "pending_count": 0,
              "overdue_count": 0,
              "bypassed_count": 0,
              "oldest_age_hours": 0.0,
              "worst_approval": "none",
          }

          doc = load_snapshot()
          if doc is not None:
              if isinstance(doc, dict):
                  approvals = doc.get("approvals")
              else:
                  approvals = doc
              if not isinstance(approvals, list):
                  approvals = []
              pending = 0
              overdue = 0
              bypassed = 0
              oldest = 0.0
              worst = "none"
              for entry in approvals:
                  if not isinstance(entry, dict):
                      continue
                  status = str(entry.get("status") or "").strip().lower()
                  age = max(0.0, as_float(entry.get("age_hours"), 0.0))
                  pipeline = str(entry.get("pipeline") or "unknown-pipeline")
                  environment = str(entry.get("environment") or "unknown-environment")
                  name = pipeline + "/" + environment
                  if status in ("auto_bypassed", "bypassed"):
                      bypassed += 1
                  elif status == "pending":
                      pending += 1
                      if age > sla_hours:
                          overdue += 1
                          if age > oldest:
                              oldest = age
                              worst = name
              verdict = "ESCALATE" if (overdue > 0 or bypassed > 0) else "CLEAR"
              outputs.update({
                  "parse_ok": True,
                  "verdict": verdict,
                  "pending_count": pending,
                  "overdue_count": overdue,
                  "bypassed_count": bypassed,
                  "oldest_age_hours": round(oldest, 1),
                  "worst_approval": worst,
              })

          print("verdict=%s pending=%d overdue=%d bypassed=%d oldest=%.1f" % (
              outputs["verdict"], outputs["pending_count"], outputs["overdue_count"],
              outputs["bypassed_count"], outputs["oldest_age_hours"]))
          print("::" + json.dumps({"outputs": outputs}) + "::")

  # --- 4. escalation gate ---
  - id: escalation_gate
    type: io.kestra.plugin.core.flow.If
    description: Branch on the audit verdict so only an overdue or auto-bypassed
      window reaches the GitHub issue, the Slack alert and the failing gate.
    condition: "{{ outputs.score_approvals.vars.verdict == 'ESCALATE' }}"
    then:
      # --- 5. escalation ---
      - id: file_overdue_issue
        type: io.kestra.plugin.github.issues.Create
        description: Open a labeled issue carrying the overdue count, the oldest age,
          the worst approval and every bypassed check so the stall lands in the
          backlog the release team already works.
        jwtToken: "{{ secret('GITHUB_TOKEN') }}"
        repository: "{{ inputs.github_repository }}"
        title: "Azure DevOps approval audit escalated for {{ inputs.project_name }}"
        body: |
          ## Azure DevOps environment approval audit

          Project **{{ inputs.project_name }}** reported **{{ outputs.score_approvals.vars.overdue_count }}** overdue pending approval(s) and **{{ outputs.score_approvals.vars.bypassed_count }}** auto-bypassed approval(s) against a {{ inputs.approval_sla_hours }} hour SLA.

          Oldest overdue age: **{{ outputs.score_approvals.vars.oldest_age_hours }} hours** on `{{ outputs.score_approvals.vars.worst_approval }}`.

          Pending approvals in the snapshot: {{ outputs.score_approvals.vars.pending_count }}.

          Execution {{ execution.id }}

          Chase the approver group for the overdue production check, review the timeout policy that auto-approved the stale check, then close this issue once the approval queue is clear.
        labels:
          - azure-devops
          - approval-gate
          - release-ops
      - id: alert_approval_stall
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Tell the release channel which approval is the oldest stall and how
          many bypasses the snapshot carried.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "text": "Azure DevOps approval audit ESCALATED for {{ inputs.project_name }} - {{ outputs.score_approvals.vars.overdue_count }} overdue pending approval(s) past the {{ inputs.approval_sla_hours }} hour SLA, oldest {{ outputs.score_approvals.vars.oldest_age_hours }} hour(s) on {{ outputs.score_approvals.vars.worst_approval }}, {{ outputs.score_approvals.vars.bypassed_count }} auto-bypassed approval(s). An issue is waiting in {{ inputs.github_repository }}. Execution {{ execution.id }}."
          }
    else:
      # --- 6. clean window ---
      - id: log_no_breach
        type: io.kestra.plugin.core.log.Log
        description: Record the clean audit so the execution history doubles as SLA
          compliance evidence.
        message: "Azure DevOps approval audit clean for {{ inputs.project_name }} - {{
          outputs.score_approvals.vars.pending_count }} pending approval(s),
          none past the {{ inputs.approval_sla_hours }} hour SLA and no
          auto-bypasses in the snapshot."

errors:
  - id: notify_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Escalate any failure to the ops channel with the execution url.
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {"text": "FAILED: {{ flow.id }} {{ execution.id }}"}

outputs:
  - id: verdict
    type: STRING
    description: Audit verdict for the window, ESCALATE when an approval is overdue
      or was auto-bypassed, CLEAR when the window is clean, UNKNOWN when the
      snapshot could not be parsed.
    value: "{{ outputs.score_approvals.vars.verdict }}"
  - id: overdue_count
    type: INT
    description: Number of pending approvals whose age exceeded the configured SLA
      hours in the scored snapshot.
    value: "{{ outputs.score_approvals.vars.overdue_count }}"
  - id: oldest_age_hours
    type: FLOAT
    description: Age in hours of the oldest overdue pending approval, or 0.0 when
      nothing breached the SLA.
    value: "{{ outputs.score_approvals.vars.oldest_age_hours }}"
  - id: bypassed_count
    type: INT
    description: Number of approvals the snapshot reported as auto-bypassed instead
      of human approved.
    value: "{{ outputs.score_approvals.vars.bypassed_count }}"
  - id: pending_count
    type: INT
    description: Total number of approvals still awaiting a decision when the
      snapshot was collected.
    value: "{{ outputs.score_approvals.vars.pending_count }}"
  - id: worst_approval
    type: STRING
    description: Pipeline and environment of the oldest overdue approval in
      pipeline/environment form, or none when nothing breached the SLA.
    value: "{{ outputs.score_approvals.vars.worst_approval }}"