Query icon
OutputValues icon
Log icon
If icon
SlackIncomingWebhook icon
Pause icon
Switch icon
Loop icon
Fail icon
Sleep icon
Schedule icon

Oracle Resumable Space Suspended Session Guard

Catch Oracle sessions frozen SUSPENDED by resumable space allocation, add space or abort the statement behind an approval, then re-check dba_resumable.

Categories
DataInfrastructure

Turn on RESUMABLE space allocation and an Oracle statement that runs out of space stops raising ORA-01653. It suspends instead. The session keeps its transaction open, keeps every table and row lock it has taken. Then it waits. Nothing fails, so nothing reaches error monitoring. Meanwhile every session that wants one of those locks queues behind a session whose wait event is "statement suspended, wait error to be cleared". The only place the cause is written down is dba_resumable.

This blueprint reads that view. It reports the sessions that have been SUSPENDED longer than a grace window you set, with the space error, the tablespace, how many locks each one is holding and how many sessions it is blocking. Then it either grows the tablespace so the statement carries on. Or it ends the statement with DBMS_RESUMABLE.ABORT so the locks are released, behind a dry-run gate plus a human approval. After either one it reads dba_resumable again and fails the run if a session is still suspended.

This blueprint was created by zkasuran.

Findings

What is read Column Why it matters
The frozen statement dba_resumable.status = 'SUSPENDED' The only record that the statement is waiting rather than failing
How long dba_resumable.suspend_time A VARCHAR2 holding MM/DD/YY HH24:MI:SS, parsed to get the age
Which space ran out dba_resumable.error_number plus error_parameter1 ORA-01653 carries the tablespace name in parameter 1
What it is holding v$lock TM and TX rows for that sid A suspended session holds its locks for as long as it waits
Who is stuck behind it v$session.blocking_session The damage the suspension is doing right now

suspend_time is a string, so it is parsed with VALIDATE_CONVERSION first and the flow falls back to v$session.last_call_et on any instance whose format differs. A parallel operation is aborted through coord_session_id where one is set, because DBMS_RESUMABLE.ABORT takes the coordinator.

Safety

An abort is not a cancel. It rolls the suspended transaction back, so everything the statement had written is gone. The application gets its error at that moment instead of hours later. Abort a session that only needed another datafile. You destroyed a load that would have finished on its own. Four things keep that from happening by accident.

  • auto_remediate is false and dry_run is true by default, so a scheduled run reads the dictionary and tells you. Both have to be flipped before anything runs.
  • grace_minutes decides what counts as frozen. A statement that pauses for a few seconds while an autoextend fires is normal, so the default 10 minutes keeps it out of the findings.
  • A Pause holds the run. The task description spells out what an abort costs, the session list is already in the log. An unanswered resume reads as refusal because the gate is approved ?? false.
  • Every abort is guarded in the database. The statement the flow runs re-reads dba_resumable inside its own block and calls ABORT only if that session id is still SUSPENDED under the same statement name. A session that freed itself in the gap between the check and the approval is left running.

excluded_resumable_names is there for the load that is meant to wait. Name it in the list and it never shows up as a finding.

How it works

  1. detect (io.kestra.plugin.jdbc.oracle.Query, read-only) joins dba_resumable to v$session, dba_users plus dba_tablespaces, keeps the rows whose status is SUSPENDED past grace_minutes, counts the TM and TX locks each session holds and the sessions it blocks. It builds both candidate fix statements per row.
  2. collect (io.kestra.plugin.core.output.OutputValues) deduplicates those statements and keeps the session id list for the re-read. summary (io.kestra.plugin.core.log.Log) logs one line per frozen session.
  3. route (io.kestra.plugin.core.flow.If):
    • clean: nothing has been suspended past the window, so it logs and stops.
    • findings: alert posts the list to Slack, because a suspended statement raises no error for anything else to pick up.
      • remediate_gate runs only when auto_remediate is true and dry_run false.
      • approve (io.kestra.plugin.core.flow.Pause) waits for a resume carrying approved.
      • pick_mode (io.kestra.plugin.core.flow.Switch) picks the branch. abort_session runs the guarded DBMS_RESUMABLE.ABORT block once per session in a Loop at concurrencyLimit: 1. add_space resizes the first datafile of each named tablespace the same way. It logs a warning instead when the space error names no growable tablespace.
      • settle (io.kestra.plugin.core.flow.Sleep) waits, because Oracle rechecks a suspended statement on a timer.
      • reverify reads dba_resumable again for those session ids. verify_result fails the run through still_suspended if any of them is still SUSPENDED, otherwise cleared logs the before and after counts.
  4. errors posts to Slack on any failure, including a session that survived the remediation.

What you get

  • A name for an outage that normally has none. Sessions pile up on locks, the application times out, nothing is in the alert log, because the session at the head of the queue is suspended rather than failed.
  • Two fixes for the same state, picked per run. Grow the tablespace when the load deserves to finish, abort when the locks matter more than the load.
  • A default posture that only reads. The write needs auto_remediate, a false dry_run plus a human answering the pause.
  • An abort that checks the session is still suspended at the moment it fires, inside the database, not in the orchestrator.
  • Proof the frozen session is gone: a re-read of dba_resumable that fails the run when one survives.

Who it's for

DBAs and platform teams running Oracle with RESUMABLE enabled, usually turned on for overnight loads so a full tablespace pauses the job instead of killing it. The pause is the feature. A pause nobody notices, holding locks until the DBA logs in, is the problem this closes.

Why orchestrate this with Kestra

Detecting it needs a dictionary read. Fixing it needs DDL or a package call, a chosen strategy, a human in the loop for something that destroys work. Then a second read to prove it worked. A cron job that calls DBMS_RESUMABLE.ABORT has no gate, no approval record plus no verification, which on a statement this destructive is the wrong tool. Kestra holds the run at a Pause, keeps the input values and the log of what was found next to the approval, then fails loudly when the dictionary disagrees with the remediation.

Tested end to end

Run against Oracle Database 23ai Free, Kestra 2.0.5, with a 8 MB tablespace with AUTOEXTEND OFF and a session that ran ALTER SESSION ENABLE RESUMABLE TIMEOUT 7200 before a bulk insert that overran it.

Path Result
Healthy instance, no suspended session 0 findings, clean
Real ORA-01653 suspension, 0 minute grace 1 finding, holding 3 locks, blocking 1 session
dry_run=true, auto_remediate=false reported, dictionary unchanged
add_space, approved datafile 8 MB to 136 MB, statement resumed, 0 suspended
abort_session, approved session aborted, locks released, 0 suspended
Pause resumed with no answer nothing changed, session still suspended
Tablespace offline before the re-read still_suspended failed the run

Every before and after was read back with sqlplus against dba_resumable, v$lock plus v$session, not from the Kestra log.

Prerequisites

Secrets

Secret Used for
ORACLE_PASSWORD Default for the password input
SLACK_WEBHOOK_URL The findings alert plus the errors alert

Inputs

Input Type Default What it does
jdbc_url STRING jdbc:oracle:thin:@//oracle-host:1521/FREEPDB1 The PDB service, not the CDB root
username STRING SYSTEM Reads the dictionary, runs the abort or the resize
password STRING {{ secret('ORACLE_PASSWORD') }} Database password
grace_minutes INT 10 How long SUSPENDED counts as frozen
remediation SELECT abort_session abort_session or add_space
add_space_mb INT 256 Megabytes added by add_space
excluded_resumable_names ARRAY __NO_EXCLUSION__ Statement names left alone
verify_delay DURATION PT20S Settle time before the re-read
auto_remediate BOOL false Allows a change
dry_run BOOL true Must be false for a change

The reads need SELECT on dba_resumable, v$session, v$lock, dba_users, dba_tablespaces plus dba_data_files, which SELECT_CATALOG_ROLE grants. abort_session needs EXECUTE on DBMS_RESUMABLE, add_space needs ALTER DATABASE. Point jdbc_url at the PDB service: dba_resumable is per container, so a CDB root connection does not see a session suspended inside a PDB.

Quick start

  1. Add the ORACLE_PASSWORD plus SLACK_WEBHOOK_URL secrets.
  2. Set jdbc_url to your PDB service and username to a user with SELECT_CATALOG_ROLE.
  3. Run it with the defaults. It only reads. The log names every session suspended past 10 minutes with the locks it holds.
  4. To clear one, pick remediation, set auto_remediate=true plus dry_run=false, run it, read the session list in the log, then resume the pause with approved=true.
  5. Enable the Schedule trigger once the report looks right. It is shipped disabled and forces report-only inputs.

How to extend

  • Raise grace_minutes to the length of your longest legitimate space pause so an autoextend that is simply slow never reads as frozen.
  • Swap the datafile RESIZE for ALTER TABLESPACE ... ADD DATAFILE where you prefer a new file over a bigger one. Use AUTOEXTEND ON MAXSIZE where the mount has room.
  • Add a branch on error_number so ORA-30036 on the undo tablespace and ORA-01536 on a quota take their own fix instead of falling through to the abort.
  • Write each finding to a KV pair keyed by session id to count how often the same job freezes, which turns a recurring suspension into a sizing decision.
  • Point the Slack alert at an incident tool so the sessions queued behind the lock get the same page a failed job would.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.