Schedule icon
Request icon
Switch icon
SlackIncomingWebhook icon
If icon
Log icon

Apache SeaTunnel Job Failure and Zeta Cluster Health Sentinel

Audit Apache SeaTunnel Zeta Engine REST API for failed streaming jobs, dispatch Slack incident alerts, and correlate cluster overview health.

Categories
DataInfrastructure

Apache SeaTunnel is a next-generation, high-performance distributed data integration engine designed to synchronize massive volumes of data across heterogeneous data sources, warehouses, and storage engines. Powered by the Zeta distributed execution engine, SeaTunnel manages streaming connectors and batch transformations spanning Kafka, ClickHouse, Apache Iceberg, Hudi, and relational databases. In production, unannounced upstream schema changes, network timeouts between source and sink connectors, or worker slot memory exhaustion can cause streaming pipelines to crash and transition into a FAILED state. When integration jobs fail, downstream tables stop receiving fresh landing data, resulting in silent data freshness SLA breaches. This blueprint provides automated continuous surveillance over the SeaTunnel Zeta REST API, detects failed jobs, alerts engineering teams via Slack, and correlates job failures with cluster worker slot metrics via GET /overview.

How it works

  1. Scheduled trigger (every_fifteen_minutes) initiates diagnostic health verification every 15 minutes, with manual runs supported via customizable inputs.
  2. Diagnostic probe (probe_failed_jobs) connects to the SeaTunnel Zeta coordinator REST API (port 8080) via io.kestra.plugin.core.http.Request and calls /finished-jobs/FAILED to inspect recently failed pipelines.
  3. Classification logic evaluates the response array using Pebble templating, discerning whether any integration jobs are in a FAILED state (CRITICAL), if all jobs are healthy (HEALTHY), or if the coordinator endpoint was unreachable (PROBE_FAILED).
  4. Routing task (route_by_job_health) switches execution along dedicated paths according to the classified severity.
  5. On CRITICAL state, structured Slack incident cards notify operations channels with coordinator endpoints, failed job counts, and severity classifications.
  6. When failed jobs are identified and inspect_cluster_overview is enabled, query_cluster_overview queries GET /overview to capture cluster-wide worker slot availability, active thread counts, and memory metrics.
  7. On PROBE_FAILED, dedicated alerts notify administrators of unreachable coordinator ports or network partitions.
  8. Audit log task (log_audit_summary) writes execution metadata and classified status to task logs.

What you get

  • Automated continuous surveillance of Apache SeaTunnel data integration pipelines across all connected sources and sinks.
  • Early detection of failed streaming jobs before downstream analytics and ML feature stores serve stale data.
  • Automated cluster diagnostics capturing worker slot allocation and coordinator health during active incidents.
  • Full execution traceability and audit logging without leaking infrastructure credentials in logs or alert payloads.
  • Clean code standards with strictly zero emojis across workflow definitions, notifications, and logs.

Who it's for

  • Data Platform Engineers and Analytics Engineers operating large-scale Apache SeaTunnel data ingestion pipelines.
  • Site Reliability Engineers ensuring continuous real-time data landing and pipeline uptime SLAs.
  • Data Infrastructure Teams deploying hybrid cloud ETL and CDC data movement architectures.

Why orchestrate this with Kestra

  • Declarative YAML definition unifies distributed pipeline monitoring, conditional error triage, and cluster diagnostics into version-controlled GitOps workflows.
  • Native HTTP request capabilities interact directly with SeaTunnel's Zeta REST API without external agent daemons or Python container overhead.
  • Built-in conditional routing isolates healthy baseline checks from multi-tiered incident notification cards.
  • Immutable execution history provides operational evidence documenting pipeline recovery timelines and recurring failure patterns.
  • Cloud-native scheduling and flow-level error-trapping ensure continuous sentinel operation independent of external cron infrastructure.

Prerequisites

  • Apache SeaTunnel cluster running with Zeta Engine REST HTTP service enabled (default port 8080).
  • Network connectivity permitting HTTP communication from Kestra to the SeaTunnel coordinator.
  • Slack incoming webhook URL for operational incident dispatch.

Secrets

  • SLACK_WEBHOOK_URL: Incoming webhook URL for posting incident cards to Slack.

Quick start

  1. Configure SLACK_WEBHOOK_URL in your Kestra namespace secrets.
  2. Deploy this blueprint to your Kestra instance and trigger an initial run with default inputs to verify connectivity against your SeaTunnel coordinator.
  3. Review the execution log in the Kestra UI to verify /finished-jobs/FAILED response parsing.
  4. Enable the every_fifteen_minutes schedule trigger to activate automated continuous monitoring.

How to extend

  • Add /running-jobs inspection to calculate elapsed execution time and flag stuck batch jobs.
  • Integrate PagerDuty alerting via io.kestra.plugin.pagerduty.notifications.PagerDutyAlert for critical real-time ingestion pipeline failures.
  • Attach automated job restart triggers using SeaTunnel CLI or API to automatically re-enqueue transiently failed jobs.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.