Hugging Face Inference Endpoint Health Sentinel for Kestra

id: huggingface-endpoint-health-sentinel
namespace: company.team
description: |
  Every 15 minutes, read the Hugging Face Inference Endpoints of a namespace
  (read-only: no endpoint is paused, resumed, scaled or deleted) and
  classify model serving as HEALTHY, DEGRADED, FAILED or UNREACHABLE. Log
  quiet runs and alert Slack when an endpoint has failed, is stuck starting
  or updating, or is running with fewer ready replicas than it should.

inputs:
  - id: stuck_minutes
    type: INT
    defaults: 30
    displayName: "Stuck Threshold (minutes)"
    description: "An endpoint still pending, initializing or updating after this
      many minutes is counted as stuck."

triggers:
  - id: health_check
    type: io.kestra.plugin.core.trigger.Schedule
    description: Check endpoint health every 15 minutes. Shipped disabled; enable it
      once the Hugging Face secrets are set and a manual run has been verified.
    cron: "*/15 * * * *"
    disabled: true

tasks:
  - id: list_endpoints
    type: io.kestra.plugin.core.http.Request
    description: List up to 100 Inference Endpoints in the namespace with their
      status from the Inference Endpoints API. This is a GET; no endpoint is
      changed.
    uri: "https://api.endpoints.huggingface.cloud/v2/endpoint/{{
      secret('HF_NAMESPACE') }}?limit=100"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('HF_TOKEN') }}"
    options:
      allowFailed: true
      timeout:
        connectTimeout: PT10S
        readIdleTimeout: PT30S
    retry:
      type: constant
      interval: PT10S
      maxAttempts: 3
    allowFailure: true

  - id: summarize_endpoints
    type: io.kestra.plugin.core.debug.Return
    description: Group endpoints by status.state, flag failed and updateFailed ones,
      flag ones stuck in a transition state for longer than the threshold, flag
      running ones with fewer ready replicas than targeted, and classify in one
      jq query. Paused and scaled-to-zero endpoints are counted but are not
      alerts. A missing or non-200 response becomes UNREACHABLE.
    format: >-
      {{ (outputs.list_endpoints.code is not defined or
      outputs.list_endpoints.code != 200) ?
      '{"status":"UNREACHABLE","total":0,"running":0,"idle":0,"failed":"none","stuck":"none","short":"none"}'
      : (fromJson(outputs.list_endpoints.body) | jq(' (.items // []) as $e | "'
      ~ (now() | dateAdd(-inputs.stuck_minutes, 'MINUTES') |
      date("yyyy-MM-dd'T'HH:mm:ss", timeZone="UTC")) ~ '" as $cutoff | [$e[] |
      select(.status.state == "failed" or .status.state == "updateFailed") |
      (.name + " (" + .status.state + ": " + ((.status.errorMessage //
      .status.message // "") | .[0:80]) + ")")] as $failed | [$e[] |
      select((.status.state == "pending" or .status.state == "initializing" or
      .status.state == "updating") and ((.status.updatedAt // "") < $cutoff)) |
      (.name + " (" + .status.state + " since " + (.status.updatedAt // "?") +
      ")")] as $stuck | [$e[] | select(.status.state == "running" and
      ((.status.readyReplica // 0) < (.status.targetReplica // 0))) | (.name + "
      (" + ((.status.readyReplica // 0) | tostring) + "/" +
      ((.status.targetReplica // 0) | tostring) + " ready)")] as $short |
      {total: ($e | length), running: ([$e[] | select(.status.state ==
      "running")] | length), idle: ([$e[] | select(.status.state == "paused" or
      .status.state == "scaledToZero")] | length), failed: (if ($failed |
      length) == 0 then "none" else ($failed | join("; ")) end), stuck: (if
      ($stuck | length) == 0 then "none" else ($stuck | join("; ")) end), short:
      (if ($short | length) == 0 then "none" else ($short | join("; ")) end),
      status: (if ($failed | length) > 0 then "FAILED" elif ($stuck | length) >
      0 or ($short | length) > 0 then "DEGRADED" else "HEALTHY" end)} ') | first
      | toJson) }}

  - id: route_by_endpoint_health
    type: io.kestra.plugin.core.flow.Switch
    description: Route to a quiet log when every endpoint is healthy, a Slack
      message when one is stuck or short of replicas, or a Slack page when one
      has failed or the API could not be read.
    value: "{{ fromJson(outputs.summarize_endpoints.value).status }}"
    cases:
      "HEALTHY":
        - id: log_healthy
          type: io.kestra.plugin.core.log.Log
          message: "Hugging Face endpoints healthy: {{
            fromJson(outputs.summarize_endpoints.value).running }} running and
            {{ fromJson(outputs.summarize_endpoints.value).idle }} paused or
            scaled to zero, out of {{
            fromJson(outputs.summarize_endpoints.value).total }}."
      "DEGRADED":
        - id: alert_degraded_endpoints
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: Post a Slack message when an endpoint is stuck in a transition
            state or running with fewer ready replicas than targeted.
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {
              "text": "Hugging Face endpoints DEGRADED. Stuck: {{ fromJson(outputs.summarize_endpoints.value).stuck }}. Short of replicas: {{ fromJson(outputs.summarize_endpoints.value).short }}. Requests may be slow or failing. Execution {{ execution.id }}."
            }
      "FAILED":
        - id: alert_failed_endpoints
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: Page Slack when an endpoint is in the failed or updateFailed state,
            meaning the model is not serving.
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {
              "text": "Hugging Face endpoints FAILED: {{ fromJson(outputs.summarize_endpoints.value).failed }}. These models are not serving; check the endpoint logs in the Hugging Face console. (Also stuck: {{ fromJson(outputs.summarize_endpoints.value).stuck }}.) Execution {{ execution.id }}."
            }
      "UNREACHABLE":
        - id: alert_unreachable_api
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: Page Slack when the Inference Endpoints API could not be read,
            indicating an outage, a wrong namespace or an invalid token.
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {
              "text": "Hugging Face endpoint sentinel UNREACHABLE: the Inference Endpoints API did not return a usable response (HTTP {{ outputs.list_endpoints.code ?? 'no response' }}). Check HF_NAMESPACE and the token. Execution {{ execution.id }}."
            }
    defaults:
      - id: log_unclassified
        type: io.kestra.plugin.core.log.Log
        message: "Hugging Face endpoint sentinel completed with an unexpected
          classification: {{ outputs.summarize_endpoints.value }}."

  - id: log_audit_result
    type: io.kestra.plugin.core.log.Log
    description: Record one audit line for every run, whichever branch fired, so
      endpoint health history stays queryable even if Slack delivery fails.
    message: "Hugging Face endpoint sentinel audit: http={{
      outputs.list_endpoints.code ?? 'n/a' }} summary={{
      outputs.summarize_endpoints.value }} execution={{ execution.id }}."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Alert Slack when the sentinel flow itself fails, so a broken check
      is never mistaken for healthy endpoints.
    url: "{{ secret('SLACK_WEBHOOK') }}"
    payload: |
      {
        "text": "Hugging Face endpoint sentinel flow {{ flow.id }} FAILED (execution {{ execution.id }}). Model serving health is now unknown; check the task logs and the Inference Endpoints console."
      }

outputs:
  - id: status
    type: STRING
    value: "{{ fromJson(outputs.summarize_endpoints.value).status }}"

Teams that serve their own models on Hugging Face Inference Endpoints depend on each endpoint staying in the running state with its replicas ready. An endpoint can drop to failed after a bad image or out-of-memory error, sit in initializing because no GPU is available, or keep running with fewer ready replicas than targeted, and every downstream app, agent or RAG pipeline that calls it starts timing out. This blueprint turns a scheduled, read-only call to the Inference Endpoints API into a model-serving gate: io.kestra.plugin.core.http.Request lists the namespace's endpoints, one jq query finds failed, stuck and under-replicated ones, and the result routes to a quiet log or a Slack alert.

How it works

list_endpoints (io.kestra.plugin.core.http.Request) calls GET /v2/endpoint/{namespace}?limit=100 with a bearer token. It is a read: no endpoint is paused, resumed, scaled or deleted. Retries cover short network blips, and allowFailed plus allowFailure keep an API error from aborting the flow so it can be reported instead.
summarize_endpoints (io.kestra.plugin.core.debug.Return) first guards for a missing or non-200 response (UNREACHABLE). Otherwise one jq query reads each endpoint's status.state (pending, initializing, updating, updateFailed, running, paused, failed, scaledToZero): failed and updateFailed endpoints are listed with their error message; pending, initializing and updating endpoints whose status.updatedAt is older than stuck_minutes are listed as stuck; running endpoints with readyReplica below targetReplica are listed as short. Any failed endpoint is FAILED, any stuck or short one is DEGRADED, otherwise HEALTHY. Paused and scaled-to-zero endpoints are counted, not alerted.
route_by_endpoint_health (io.kestra.plugin.core.flow.Switch) branches on that classification: HEALTHY logs quietly, DEGRADED posts a Slack message, FAILED and UNREACHABLE page Slack. A defaults fallback logs anything unexpected.
log_audit_result (io.kestra.plugin.core.log.Log) writes one audit line per run with the HTTP status and the full summary.
The flow-level errors block pages Slack if the flow itself fails, and the health_check trigger runs every 15 minutes, shipped disabled: true.

What you get

A read-only model-serving check across every endpoint in a namespace, with no custom scripting and no extra plugin to install.
A status output (HEALTHY / DEGRADED / FAILED / UNREACHABLE) other flows can gate on, for example to route an agent to a fallback model while the primary endpoint is down.
Slack messages that name each affected endpoint with its state, replica count or error message.
An audit line on every run, separate from Slack.

Who it's for

ML and platform teams serving open models (LLMs, embeddings, rerankers) on Hugging Face Inference Endpoints.
Teams whose agents or RAG pipelines in Kestra call those endpoints and need to know when the model behind them is unhealthy.
Anyone paying for GPU endpoints who wants to notice a stuck deployment before the bill or the users do.

Why orchestrate this with Kestra

The Inference Endpoints console shows state when someone opens it, and its notifications cover a few events. Kestra turns a plain, read-only API call into a scheduled, typed classification across the whole namespace with retries, a failure path and an execution history, right next to the AI flows that depend on those endpoints.

Prerequisites

A Hugging Face account or organization with at least one Inference Endpoint.
A fine-grained access token with read access to Inference Endpoints for that namespace.
A Slack incoming webhook for the DEGRADED, FAILED, UNREACHABLE and flow-failure alerts.

Secrets

HF_NAMESPACE: the user or organization name that owns the endpoints.
HF_TOKEN: Hugging Face access token with Inference Endpoints read permission.
SLACK_WEBHOOK: Slack incoming webhook used for every alert branch and the flow-failure alert.

In Kestra OSS, secrets are supplied as environment variables prefixed SECRET_ and base64-encoded, for example SECRET_HF_NAMESPACE=$(echo -n 'my-org' | base64), and read back with {{ secret('HF_NAMESPACE') }}. Use the Enterprise secrets backend for encryption at rest and access control.

Quick start

Create a small CPU Inference Endpoint (for example a sentence-transformers model) from the Model Catalog.
Add HF_NAMESPACE, HF_TOKEN and SLACK_WEBHOOK as secrets.
Run the flow while the endpoint is initializing and again once it is running, and compare the logged states with the console.
Enable health_check.

How to extend

Follow nextCursor when the namespace has more than 100 endpoints.
Add a Request to GET /v2/endpoint/{namespace}/{name}/logs for each failed endpoint and attach the last log lines to the Slack message.
Filter with the tags query parameter to watch only production endpoints.
Store the last status per endpoint in a KV store and alert only on changes.

Inputs

stuck_minutes: type INT, default 30.

Outputs

status: one of HEALTHY, DEGRADED, FAILED, UNREACHABLE.
Example path: {{ fromJson(outputs.summarize_endpoints.value).failed }} for the failed endpoints and their error messages.

Pitfalls

"Stuck" is measured from status.updatedAt in UTC; a large model on a scarce GPU can legitimately take longer than 30 minutes to initialize, so raise stuck_minutes for those.
An endpoint that scales to zero on purpose shows scaledToZero and is not an alert; the first request after that wakes it up with a cold-start delay.
Only the first 100 endpoints are read unless you add cursor paging.
This flow only reads; it never pauses, resumes, scales or deletes an endpoint.