Schedule icon
Request icon
Script icon
Switch icon
Log icon
SlackIncomingWebhook icon

Railway Deployment Crash Loop & OOM Sentinel

Audit Railway service deployments on a schedule to detect crashed containers and OOM failures before application downtime impacts live users.

Categories
CloudInfrastructure

Modern cloud deployment platforms like Railway allow engineering teams to continuously build and ship containerized microservices, background workers, and web APIs with automated Git push triggers. However, during runtime traffic surges, memory allocation leaks, or misconfigured runtime variables, services can crash immediately post-deployment or enter silent crash-restart loops. If engineers do not actively monitor the Railway dashboard, degraded services can remain offline for hours before customer complaints surface.

This blueprint implements an automated Day 2 operations sentinel for Railway projects. Operating on a periodic schedule, it directly queries the Railway GraphQL API v2, inspects the latest deployment status across all project services, detects CRASHED or FAILED containers, and dispatches rich Slack incident alert cards with actionable recovery commands.

How it works

  1. Scheduled Invocation: The workflow executes on a periodic schedule (0 */2 * * *, every 2 hours, shipped disabled for initial configuration) or on demand.
  2. Railway GraphQL Query: fetch_railway_deployments executes an HTTP POST request to https://backboard.railway.app/graphql/v2 via io.kestra.plugin.core.http.Request, authenticating with the bearer token secret RAILWAY_API_TOKEN.
  3. Python Metric Extraction: parse_deployment_health executes inside an isolated python:3.11-slim container, parsing the GraphQL payload. It inspects recent deployments across all services and checks for CRASHED, FAILED, or REMOVED states.
  4. Operational Branching: evaluate_deployment_status utilizes io.kestra.plugin.core.flow.Switch to evaluate the classified status:
    • When Healthy: Emits an audit log confirming all services are online.
    • When Degraded: Invokes io.kestra.plugin.slack.notifications.SlackIncomingWebhook to dispatch a formatted alert card to #devops-alerts detailing the offending service name, deployment UUID, and suggested CLI redeploy commands.
  5. Telemetry Logging: log_audit_completion records the overall audit summary and failed service counts.
  6. Error Resilience: A dedicated errors handler catches HTTP connection timeouts or authentication failures, ensuring on-call teams are alerted if the health check itself cannot reach the API.

What you get

  • Automated Crash Loop Detection: Continuous inspection catches container crashes, build failures, and memory OOM terminates within minutes.
  • Zero Infrastructure Footprint: Runs completely serverless using Kestra native tasks and containerized Python parsing, requiring no persistent monitoring agents.
  • Actionable Slack Alerts: Context-rich Slack messages direct SRE and platform engineers immediately to the offending service with concrete railway redeploy commands.
  • Multi-Service Visibility: Inspects all microservices and background workers across an entire Railway project in a single unified audit pass.

Who it's for

  • Platform Engineers & DevOps Teams managing microservice fleets and web applications deployed on Railway.
  • Full-Stack Developers operating production apps who need automated uptime safeguards without dedicated monitoring stacks.
  • Site Reliability Engineers (SREs) maintaining infrastructure reliability and container lifecycle management.

Why orchestrate this with Kestra

Writing custom cron jobs or Lambda functions to poll GraphQL APIs creates unmonitored scripts, unmasked secrets, and brittle alert plumbing. Kestra provides:

  • Declarative YAML Orchestration: The entire monitoring and alert topology is codified in version-controlled YAML with transparent execution history.
  • Encrypted Secret Management: Railway API tokens and Slack incoming webhook URLs remain securely encrypted and masked in all execution logs.
  • Built-in Flow Logic: Native Switch branching and containerized Python scripts provide robust error recovery without external microservice dependencies.

Prerequisites

  • A Railway account and project with an API token from railway.app/account/tokens.
  • An incoming Slack webhook URL for operational alerting.

Secrets

Configure the following secrets in your Kestra namespace or instance settings:

Secret Name Description Example
RAILWAY_API_TOKEN Railway API personal access token or project token ry_live_••••••••••••
SLACK_WEBHOOK Incoming webhook URL for destination Slack channel https://hooks.slack.com/services/T00/B00/X00

Quick start

  1. In Kestra, navigate to Flows and import this blueprint into your designated namespace (e.g., company.cloud).
  2. Add your RAILWAY_API_TOKEN and SLACK_WEBHOOK into Settings > Secrets.
  3. Click Execute and provide your railway_project_id to test the audit pass.
  4. Review execution logs to verify deployment parsing and ensure the Slack alert card renders properly.
  5. In the flow definition, toggle disabled: false on periodic_railway_audit to activate the automated 2-hour monitoring schedule.

How to extend

  • Automated Rollback Trigger: Chain an automated rollback task using io.kestra.plugin.core.http.Request to trigger the Railway deployment rollback endpoint when remediation_mode == 'AUTO_ROLLBACK'.
  • PagerDuty Escalation: Add an io.kestra.plugin.pagerduty.Incident task within the DEPLOYMENTS_DEGRADED branch to escalate persistent production outages to on-call engineers.
  • Multi-Project Inspection: Replace the single project input with io.kestra.plugin.core.flow.Loop to audit all organization projects in parallel.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.