Schedule icon
Request icon
Return icon
Switch icon
Log icon
SlackIncomingWebhook icon
Pause icon
If icon
Loop icon

DHIS2 Scheduled Job Failure Sentinel

Check DHIS2 scheduled jobs, alert Slack when analytics, sync or import jobs fail or get stuck, and re-run them through the DHIS2 API once someone approves.

Categories
BusinessData

DHIS2 is the health information system used by ministries of health and NGOs in more than 80 countries for routine reporting, disease surveillance and programme data. Its dashboards are only as fresh as its background jobs: the analytics tables export, data synchronisation with other instances, metadata sync, data imports and notifications. When one of these fails, DHIS2 records it in the scheduler, but dashboards keep showing yesterday's numbers and nobody is paged. This blueprint turns two read-only DHIS2 API calls into a scheduled job health gate: /api/system/info proves the server is answering and gives its own clock, /api/jobConfigurations returns every job with its last result, a single jq query counts failed and stuck jobs, and the result routes to a quiet log, an informational Slack note or a Slack page. When jobs failed, the execution then pauses for the person on call: one approval re-runs them through the DHIS2 API, and no answer means nothing is touched.

How it works

  1. read_system_info (io.kestra.plugin.core.http.Request) reads /api/system/info with basic authentication. Only an answer with a DHIS2 version counts as a working server, and its serverDate becomes the reference clock. Every request is a GET with a timeout, three retries, allowFailed: true and allowFailure: true, so an outage never aborts the flow.
  2. fetch_job_configurations reads /api/jobConfigurations with only the fields the check needs: jobType, enabled, schedulingType, jobStatus, lastExecutedStatus and lastExecuted. A runIf skips it when /system/info did not answer.
  3. classify_jobs (io.kestra.plugin.core.debug.Return) checks both responses (a missing response, a non-200 code or a body that is not JSON is UNREACHABLE), then, against the server's own serverDate, counts the jobs whose last run FAILED within lookback_hours, the enabled scheduled jobs among them, and the jobs still RUNNING after stuck_hours. A failed enabled scheduled job, a stuck job, or at least critical_failed failures is CRITICAL; any other failure is WARNING; otherwise HEALTHY.
  4. route_by_job_status (io.kestra.plugin.core.flow.Switch) logs HEALTHY quietly, posts a Slack note for WARNING, pages Slack for CRITICAL and UNREACHABLE, and logs any unexpected value in defaults.
  5. When failed jobs were found, wait_for_retry_approval (io.kestra.plugin.core.flow.Pause) waits up to approval_wait for someone to resume the execution with retry set to true or false. The Slack alert names the jobs and the execution to resume.
  6. retry_if_approved (io.kestra.plugin.core.flow.If) re-runs the jobs only on an explicit true: retry_each_failed_job (io.kestra.plugin.core.flow.Loop, one job at a time, at most max_retry_jobs, most recent first) calls POST /api/jobConfigurations/{id}/execute, the same as Run now in the DHIS2 scheduler, and alert_retry_requested posts each job's HTTP answer to Slack. A false answer or a pause that times out goes to log_retry_not_approved, and nothing is re-run.
  7. log_audit_result writes one audit line per run with both HTTP codes, the classification and the retry answer. The flow-level errors block alerts Slack if the flow fails outright, and the job_check trigger runs every hour and ships disabled: true.

What you get

  • A scheduled check of every DHIS2 background job, using only the Web API.
  • A human-approved retry: failed jobs are re-run only after someone on call says yes, and the result is posted back to Slack.
  • A status output (HEALTHY / WARNING / CRITICAL / UNREACHABLE) other flows can gate on, and a jobs JSON output with jobs, failed, failed_scheduled, stuck, the job types that failed, dhis2_version and server_date.
  • Slack messages that group failures by job type, for example DATAVALUE_IMPORT (8), and name scheduled jobs and stuck jobs.
  • An audit log line on every run, so job health history survives a Slack outage.

Who it's for

  • Health information system administrators and HMIS teams running DHIS2 for a country, state or programme.
  • Implementing partners who maintain DHIS2 instances and their synchronisation with other systems.
  • Data platform teams already using Kestra who want the same alerting for their DHIS2 servers.

Why orchestrate this with Kestra

DHIS2 keeps the result of each job, but only an administrator opening the scheduler sees it, and a failed analytics run is easy to miss until a programme manager asks why the dashboard has not moved. Kestra runs the same read-only check on a schedule, classifies all jobs in one typed status, keeps every run in the execution history, routes it to the right channel, and then holds the execution open for a person to decide on the retry. The approval, the re-run and its answer all land in the same execution, so the audit trail of who fixed what stays in one place.

Prerequisites

  • A DHIS2 instance whose Web API is reachable from Kestra (tested on DHIS2 2.43).
  • A dedicated DHIS2 user with the authority to view the scheduler (job configurations) and, for the retry, to run jobs.
  • A Slack incoming webhook for the WARNING, CRITICAL, UNREACHABLE and flow-failure alerts.

Secrets

  • DHIS2_USERNAME and DHIS2_PASSWORD: credentials of the monitoring user (HTTP basic authentication).
  • SLACK_WEBHOOK: Slack incoming webhook used for every alert branch and the flow-failure alert.

In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_ and base64-encoded, then read in flows with {{ secret('NAME') }}, for example SECRET_DHIS2_PASSWORD=$(echo -n '<password>' | base64).

Quick start

  1. To try it without your own instance, point dhis2_url at the public DHIS2 demo listed on https://play.dhis2.org (it redirects to the current stable version) with the demo credentials published there. The demo is reset regularly, so it may or may not have failed jobs when you try it.
  2. Add DHIS2_USERNAME, DHIS2_PASSWORD and SLACK_WEBHOOK as secrets and run the flow once.
  3. When the run pauses, open the execution in Kestra and resume it with retry set to true or false. Lower lookback_hours or raise critical_failed to see the other branches, then point the flow at your own instance.
  4. Enable job_check once pointed at the real server.

How to extend

  • Filter jobType to the jobs that matter most (ANALYTICS_TABLE, META_DATA_SYNC, DATA_SYNC) and send each its own channel.
  • Read /api/jobConfigurations/{id}/errors for the failed jobs and include the first error in the Slack message.
  • Store status in the KV store and alert only on changes.
  • Run a second check a few minutes after an approved retry and post whether the jobs completed.

Inputs

  • dhis2_url: type STRING, default "https://dhis2.example.org".
  • lookback_hours: type INT, default 24.
  • stuck_hours: type INT, default 6.
  • critical_failed: type INT, default 3.
  • approval_wait: type DURATION, default PT2H.
  • max_retry_jobs: type INT, default 5.

Outputs

  • status: one of HEALTHY, WARNING, CRITICAL, UNREACHABLE.
  • jobs: the full classification as JSON, for example {"dhis2_version":"2.43.2","jobs":19,"failed":8,"failed_scheduled":0,"stuck":0,"failed_jobs":"DATAVALUE_IMPORT (8)","retry_ids":["..."],"status":"CRITICAL",...}.

Links

Pitfalls

  • DHIS2 returns job times in the server's local time without an offset. The flow measures every window against the server's own serverDate, so do not compare them with Kestra's clock if you extend it.
  • One-off jobs such as DATAVALUE_IMPORT are kept as disabled job configurations after they run. Their failures count as WARNING (or CRITICAL above critical_failed); a failed enabled scheduled job is always CRITICAL.
  • jobStatus and lastExecuted only describe the most recent run. A job that failed and then succeeded on its next run is reported as healthy, by design.
  • The only write is the approved re-run. The flow never enables, edits or deletes jobs, and a declined or unanswered pause re-runs nothing.
  • A job that is still running when the retry is approved answers HTTP 409; the message shows the code, and the next check reports its outcome.
  • A re-run reruns the job as configured. If the cause (for example a missing remote server for data sync) is still there, it fails again and the next check reports it.
See How

New to Kestra?

Use blueprints to kickstart your first workflows.