TorchServe Model Server Health Sentinel

id: torchserve-model-server-health-and-inference-worker-sentinel
namespace: company.platform

description: "Continuously audits TorchServe model server runtime health and
  inference endpoints, monitors registered PyTorch model availability, alerts
  via Slack, and prevents AI inference prediction outages."

inputs:
  - id: torchserve_host
    type: STRING
    defaults: "localhost"
    description: "Hostname or IP address of the TorchServe model server instance."

  - id: inference_port
    type: INT
    defaults: 8080
    description: "HTTP port for the TorchServe inference and health ping service
      (default: 8080)."

  - id: management_port
    type: INT
    defaults: 8081
    description: "HTTP port for the TorchServe management and model registry API
      (default: 8081)."

  - id: dry_run
    type: BOOL
    defaults: true
    description: "Safety gate: when true, audits model health and logs diagnostic
      metrics without triggering automated model unregister or worker restarts."

  - id: slack_channel
    type: STRING
    defaults: "#ai-ops-alerts"
    description: "Target Slack channel for TorchServe model server outages and
      worker degradation alerts."

triggers:
  - id: periodic_torchserve_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/15 * * * *"
    disabled: true

tasks:
  - id: probe_server_ping
    type: io.kestra.plugin.core.http.Request
    description: "Polls TorchServe /ping endpoint on inference port to verify daemon
      responsiveness and core Java-Python bridge health."
    uri: "http://{{ inputs.torchserve_host }}:{{ inputs.inference_port }}/ping"
    method: GET
    options:
      timeout:
        readIdleTimeout: PT15S

  - id: probe_registered_models
    type: io.kestra.plugin.core.http.Request
    description: "Queries /models endpoint on management port to inspect registered
      PyTorch model artifacts and active worker allocations."
    uri: "http://{{ inputs.torchserve_host }}:{{ inputs.management_port }}/models"
    method: GET
    options:
      timeout:
        readIdleTimeout: PT15S

  - id: classify_server_health
    type: io.kestra.plugin.core.flow.Switch
    description: "Categorizes TorchServe status into SERVER_UNHEALTHY,
      NO_MODELS_LOADED, or HEALTHY based on health ping and model catalog
      responses."
    value: >-
      {% if outputs.probe_server_ping.code != 200 or not ('"status": "Healthy"'
      in outputs.probe_server_ping.body) %} SERVER_UNHEALTHY {% elif
      outputs.probe_registered_models.code == 200 and
      (fromJson(outputs.probe_registered_models.body).models | length) == 0 %}
      NO_MODELS_LOADED {% else %} HEALTHY {% endif %}
    cases:
      SERVER_UNHEALTHY:
        - id: alert_server_unhealthy_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: "Dispatches critical incident notification when TorchServe health
            probe reports service degradation or error code."
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "channel": "{{ inputs.slack_channel }}",
              "text": "[CRITICAL] TorchServe Model Server Health Check Failed",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*[CRITICAL] TorchServe Model Server Outage*\n*Host:* `http://{{ inputs.torchserve_host }}:{{ inputs.inference_port }}`\n*HTTP Code:* `{{ outputs.probe_server_ping.code }}`\n*Ping Response:* `{{ outputs.probe_server_ping.body | trim }}`\n*Impact:* PyTorch model inference requests are failing across downstream AI applications."
                  }
                }
              ]
            }

      NO_MODELS_LOADED:
        - id: alert_no_models_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: "Dispatches operational warning when TorchServe daemon is running
            but zero inference models are loaded in memory."
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "channel": "{{ inputs.slack_channel }}",
              "text": "[WARNING] TorchServe Running with Zero Loaded Models",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*[WARNING] TorchServe Model Catalog Empty*\n*Host:* `http://{{ inputs.torchserve_host }}:{{ inputs.management_port }}`\n*Active Models:* 0\n*Action:* Verify whether recent model store deployments completed or workers encountered GPU memory errors."
                  }
                }
              ]
            }

        - id: check_dry_run_remediation
          type: io.kestra.plugin.core.flow.If
          description: "Safeguard check ensuring diagnostic logging only triggers
            according to dry-run configuration."
          condition: "{{ inputs.dry_run == true }}"
          then:
            - id: log_dry_run_notice
              type: io.kestra.plugin.core.log.Log
              message: "Dry-run active: empty model catalog detected without attempting
                automated model registration."
          else:
            - id: log_live_remediation_intent
              type: io.kestra.plugin.core.log.Log
              message: "Live mode: empty model catalog recorded on {{ inputs.torchserve_host
                }}. Operational alert dispatched."

      HEALTHY:
        - id: log_healthy_state
          type: io.kestra.plugin.core.log.Log
          description: "Logs nominal health when server responds healthy and active models
            are registered in memory."
          message: "TorchServe Model Server is healthy on {{ inputs.torchserve_host }}:{{
            inputs.inference_port }}. Active models registered: {{
            (fromJson(outputs.probe_registered_models.body).models | length) ??
            0 }}."

  - id: log_sentinel_audit_summary
    type: io.kestra.plugin.core.log.Log
    description: "Emits immutable audit record documenting TorchServe server status
      and execution lineage."
    message: "TorchServe sentinel audit completed for http://{{
      inputs.torchserve_host }}:{{ inputs.inference_port }}. Status: {{
      outputs.classify_server_health.value | trim }} | Ping Code: {{
      outputs.probe_server_ping.code }} | Execution: {{ execution.id }}"

errors:
  - id: alert_torchserve_network_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: "Dispatches emergency Slack alert if the TorchServe container is
      offline, network-partitioned, or unreachable."
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "channel": "{{ inputs.slack_channel }}",
        "text": "[CRITICAL] TorchServe Server Connection Refused",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "*[CRITICAL] TorchServe Server Unreachable*\nFailed to connect to TorchServe instance at `http://{{ inputs.torchserve_host }}:{{ inputs.inference_port }}`. The model server daemon may have crashed, GPU memory exhausted, or container terminated."
            }
          }
        ]
      }

TorchServe is an open-source, performant model serving framework for PyTorch developed by AWS and the PyTorch Foundation. Designed for production MLOps and deep learning platforms, TorchServe delivers low-latency model inference, multi-model serving, batch prediction, and management APIs. In production deep learning environments, unhandled CUDA Out-Of-Memory (OOM) exceptions, model initialization timeouts, or memory leaks within Python backend workers can cause inference worker processes to terminate abruptly, leading to failed prediction requests across downstream applications. This blueprint deploys an automated operational sentinel that continuously verifies TorchServe daemon responsiveness, audits registered PyTorch model readiness, routes structured incident alerts to Slack, and maintains execution audit lineage.

How it works

periodic_torchserve_audit (io.kestra.plugin.core.trigger.Schedule) initiates scheduled monitoring every 15 minutes. Shipped disabled so initial connectivity can be verified manually.
probe_server_ping (io.kestra.plugin.core.http.Request) invokes GET /ping on the inference port to confirm model server daemon uptime and JVM-Python worker communication.
probe_registered_models (io.kestra.plugin.core.http.Request) invokes GET /models on the management port to inspect loaded PyTorch model artifacts and worker allocations.
classify_server_health (io.kestra.plugin.core.flow.Switch) evaluates probe results:
- SERVER_UNHEALTHY: Fires if /ping returns non-200 or unhealthy status, dispatching a high-severity incident notification via io.kestra.plugin.slack.notifications.SlackIncomingWebhook.
- NO_MODELS_LOADED: Fires when the server is running but zero models are registered in memory, alerting the MLOps team of potential deployment errors.
- HEALTHY: Confirms that both daemon runtime and model inference pipelines operate nominally.
log_sentinel_audit_summary (io.kestra.plugin.core.log.Log) writes an immutable execution log recording evaluated status codes and audit lineage.
Global errors handler catches network connection drops, timeouts, or container crashes, firing an emergency outage alert to Slack.

What you get

Automated continuous surveillance of TorchServe model server availability and inference endpoint health.
Proactive alerts on Slack before GPU memory exhaustion or worker crashes impact production AI predictions.
Safe dry-run guarding: default dry_run: true prevents automated mutations during cluster audits.
Centralized audit lineage within Kestra tracking model server availability history over time.

Who it's for

MLOps Engineers and AI Platform Teams managing PyTorch model inference serving on Kubernetes or Docker.
SREs and DevOps Teams supporting production deep learning pipelines requiring uptime guarantees.
Data Scientists deploying PyTorch models seeking automated health and worker availability alerts.

Why orchestrate this with Kestra

TorchServe exposes REST management and inference endpoints but lacks built-in alerting, cron scheduling, and multi-channel incident escalation. Relying on custom shell scripts or ad-hoc cron jobs introduces maintenance overhead, hard-coded credentials, and zero execution visibility. Kestra provides a declarative YAML blueprint featuring scheduled triggers, masked secret management, conditional branching, and comprehensive execution observability.

Prerequisites

A running TorchServe instance (version 0.8.0+) accessible over HTTP. TorchServe provides an official open-source Docker image (pytorch/torchserve:latest-cpu) requiring zero commercial licensing.
An incoming Slack webhook URL for operational notifications.

Secrets

SLACK_WEBHOOK_URL: Slack incoming webhook URL for operational notifications and incident routing.

Quick start

Start a local TorchServe test container:

docker run -d --name torchserve-test -p 8080:8080 -p 8081:8081 pytorch/torchserve:latest-cpu torchserve --start --model-store /home/model-server/model-store

Configure SLACK_WEBHOOK_URL in your Kestra namespace secrets.
Import this blueprint and execute with default inputs (dry_run: true).
Confirm that log_healthy_state records a successful health check response (status: Healthy).
Test alerting by stopping the container (docker stop torchserve-test) and re-running the flow to verify alert_torchserve_network_failure.
Enable periodic_torchserve_audit to activate recurring 15-minute surveillance.

How to extend

Add synthetic prediction probes using POST /predictions/{model_name} to benchmark inference latency against SLA thresholds.
Integrate with PagerDuty via io.kestra.plugin.pagerduty.notifications.PagerDutyAlert for high-severity model serving outages.
Export model worker memory metrics to external Prometheus dashboards using io.kestra.plugin.prometheus.

TorchServe Model Server Health & Inference Worker Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

TorchServe Model Server Health & Inference Worker Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More AI Blueprints

Generate an SEO summary of an article using HTTP REST API and OpenAI plugins

GCP Vertex AI Endpoint Traffic Split Discrepancy Guard

AI GitHub issue duplicate detector

AI enrichment example: Getting started with Kestra

Feast Feature Store Offline-to-Online Sync Drift & Parity Guard

Firecrawl Web Scraping Job Error and Stagnation Sentinel

New to Kestra?