AWS Glue Stuck Job Run Sentinel

A Glue job that fails errors out quickly and loudly. A Glue job that is merely stuck - stalled on a slow partition, waiting on a resource it will never get, looping without making progress - does neither. It sits in RUNNING consuming DPUs and billable time while nothing downstream of it moves forward, and Glue's own job timeout is often set far looser than what any individual run should actually take. This blueprint reads a specific run's status and execution time through GetJobRun, flags it the moment it either ends badly or simply runs too long, alerts Slack, and can stop it cleanly with StopJobRun - leaving any run that already reached a terminal state completely untouched.

How it works

1. periodic_job_run_check (io.kestra.plugin.core.trigger.Schedule) runs every 30 minutes, always forcing auto_remediate: "false" and dry_run: "true" through its own inputs: override regardless of this flow's defaults. Shipped disabled so you can validate a manual run first.
2. probe_job_run_status (io.kestra.plugin.aws.glue.GetJobRun, allowFailure: true) reads run_id's state, executionTime, and related fields.
3. evaluate_run_health (io.kestra.plugin.core.debug.Return) classifies the result as UNREACHABLE (state undefined), AT_RISK (a non-success terminal state, or still running/starting/stopping past max_running_minutes), or HEALTHY.
4. route_by_health (io.kestra.plugin.core.flow.Switch) branches on that classification:
   • HEALTHY: log_healthy records the current, on-budget state.
   • AT_RISK: a nested route_remediation (Switch on auto_remediate) either considers remediation ("true", gated again by dry_run through check_dry_run) or only logs that remediation is disabled ("false"). When both gates allow it, stop_stuck_job_run (io.kestra.plugin.aws.glue.StopJobRun) stops the run. alert_at_risk always fires to Slack regardless of which path ran.
   • UNREACHABLE: alert_unreachable pages Slack directly - a broken API call must never read as "job is fine".
5. log_audit_complete always runs last, printing the classification regardless of which branch fired.
6. The errors block alerts Slack separately if the flow itself fails outside these handled branches.

What you get

• A direct, Glue-native answer to "is this specific run actually healthy right now", distinguishing a genuinely stuck run from one that is simply still working within its expected time budget.
• Remediation that only ever stops the one audited run; no other run of the same job, and no other job, is ever touched.
• A three-way classification (HEALTHY/AT_RISK/UNREACHABLE) and a final status log every run, so the execution history doubles as a job-reliability trend for job_name.

Who it's for

• Data engineering teams running AWS Glue ETL jobs who need a cost and pipeline-latency safeguard against a run that quietly stalls instead of failing outright.
• Platform teams who want a standard post-launch check after triggering a critical Glue job from another Kestra flow (such as this repo's own aws-glue-athena-dynamodb.yaml), auditing the run it started.
• Anyone evaluating the Glue plugin who wants a worked example of GetJobRun and StopJobRun used together as a genuine operational safeguard.

Why orchestrate this with Kestra

Glue's own per-job timeout is a single blunt ceiling set once at the job definition level; it does not know that this particular run, for this particular input, should have finished in ten minutes rather than the two hours the job's timeout allows. Kestra supplies the schedule, the time-budget classification as a first-class branch, two independent authorization gates before anything is stopped, and an execution history that shows exactly when a run first crossed its budget and whether it was stopped.

Prerequisites

• An AWS account (or a LocalStack instance - GetJobRun/StopJobRun both expose endpointOverride for this) with a Glue job and a run ID to audit.
• IAM permissions for glue:GetJobRun and, if remediation runs, glue:StopJobRun.
• A Slack incoming webhook for alerts.

Local testing: start LocalStack with docker run -d --name glue-sentinel -p 4567:4566 localstack/localstack (host port 4567 chosen to avoid any collision with Kestra's own UI on 8080), docker network connect YOUR_KESTRA_NETWORK glue-sentinel (check docker network ls first; only needed if the Kestra Worker runs in a separate Docker network), then set endpointOverride to http://glue-sentinel:4566 (in-network) or http://localhost:4567 (from the host) on both probe_job_run_status and stop_stuck_job_run. Whether LocalStack's Community edition mocks Glue job-run state transitions realistically enough to exercise AT_RISK has not been verified in this environment; a real AWS account with a real Glue job remains the most reliable way to validate this flow end to end.

Secrets

• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_DEFAULT_REGION: AWS credentials used by both GetJobRun and StopJobRun, matching this plugin's own documented connection properties (see build-aws-ecr-image.yaml).
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_at_risk, alert_unreachable, and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded, and read back in flows with {{ secret('NAME') }} - for example SECRET_AWS_ACCESS_KEY_ID=$(echo -n '<your-access-key>' | base64). This keeps credentials out of the flow YAML but, per Kestra's own documentation, offers no encryption at rest or access control beyond the host environment; use the Enterprise secrets backend for stronger guarantees.

Inputs

• job_name (STRING, no default): Glue job audited.
• run_id (STRING, no default): specific run audited.
• max_running_minutes (INT, default 120): time budget before a still-running state counts as stuck.
• auto_remediate (SELECT: "false", "true"; default "false"): must be "true" for StopJobRun to even be considered.
• dry_run (SELECT: "true", "false"; default "true"): must be explicitly "false", together with auto_remediate: "true", for the run to actually be stopped.

Outputs

• outputs.probe_job_run_status.state / .executionTime / .startedOn / .completedOn / .errorMessage: the audited run's own fields for every run.
• outputs.evaluate_run_health.value: HEALTHY, AT_RISK, or UNREACHABLE for every run.

Quick start

1. Start a Glue job (or point at an existing one) and note its job_name and a run_id to audit - for example, the jobRunId output from this repo's own aws-glue-athena-dynamodb.yaml after triggering it.
2. Add the AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually against a run that completed successfully and confirm log_healthy fires.
4. Point run_id at a run that failed and re-run; confirm alert_at_risk fires with state: FAILED, and that log_remediation_disabled logs without issuing a stop.
5. Lower max_running_minutes below a currently-running run's actual elapsed time, run once with auto_remediate: "true" and dry_run: "true", and confirm log_dry_run_remediation describes the stop without issuing it.
6. Run once with auto_remediate: "true" and dry_run: "false" against that same still-running job and confirm stop_stuck_job_run transitions it toward STOPPED.
7. Enable periodic_job_run_check once you trust the check; it always runs in the safe auto_remediate: "false" / dry_run: "true" mode regardless of what you leave the flow's own defaults set to.

How to extend

• Chain this flow directly after a StartJobRun task (via a Subflow call passing through the new run's jobRunId) for same-execution stuck-run protection on a critical pipeline, instead of waiting for the next scheduled sweep.
• Loop over several job_name/run_id pairs with io.kestra.plugin.core.flow.Loop to audit every currently-running job across a Glue environment in one pass.
• Surface errorMessage (returned by GetJobRun when present) in the Slack alert for terminal-state failures, giving the on-call engineer Glue's own failure reason without opening the AWS console.
• Route the remediation branch through a Pause approval task instead of (or alongside) the static auto_remediate input, for a human-reviewed stop on a production job.

Pitfalls

• GetJobRun's runId and StopJobRun's jobRunId are different property names for the same value. Verified from each task's own documented schema; this blueprint reads inputs.run_id into both, but a direct copy-paste between the two tasks without checking the property name would silently fail schema validation.
• AWS Glue's full JobRunState enum has ten values, not just the two Kestra's own task description mentions as examples (SUCCEEDED, RUNNING). Verified against AWS's own Glue API reference: STARTING, RUNNING, STOPPING, STOPPED, SUCCEEDED, FAILED, TIMEOUT, ERROR, WAITING, EXPIRED. This blueprint's classification explicitly lists every non-success terminal state it treats as AT_RISK, rather than assuming only FAILED matters.
• StopJobRun has nothing to stop a run that already reached a terminal state. This blueprint only reaches stop_stuck_job_run from the AT_RISK branch, and a run classified AT_RISK purely for being in a terminal state (FAILED/TIMEOUT/ERROR/EXPIRED/STOPPED) will still have StopJobRun called against it by this flow's current logic - per Glue's own API behavior this is a safe no-op, but it does mean the task runs even when there is nothing left to stop; see How to extend for narrowing this to only the still-running case.
• Whether LocalStack Community edition realistically mocks Glue job-run state transitions has not been verified in this environment. endpointOverride is a real, documented property on both tasks that supports pointing at LocalStack, but exercising AT_RISK end to end was not done here.
• allowFailure: true is what lets UNREACHABLE be classified instead of failing the execution outright. A base Task property, not specific to this plugin; without it, an invalid job_name/run_id or a missing IAM permission would abort the flow before evaluate_run_health ever runs.
• auto_remediate and dry_run are independent gates, not a single boolean. Both must be "true"/"false" respectively for StopJobRun to run; setting only one leaves the other still blocking.
• Switch case keys are quoted strings matching the classifier's exact output. "HEALTHY", "AT_RISK", and "UNREACHABLE" must match evaluate_run_health.value verbatim; the defaults: block exists specifically to catch any value that does not match one of those three cases.

Links

• plugin-aws Glue documentation
• AWS Glue GetJobRun API reference
• AWS Glue JobRunState reference
• Kestra secrets (OSS vs Enterprise)