Replace a wedged Terraform resource that plan reports as clean

id: terraform-degraded-resource-replace-guard
namespace: company.team

description: |
  Catch the resource Terraform cannot see is broken. A container is still
  RUNNING, so the Docker provider reads it back exactly as the state file
  describes it and terraform plan prints "No changes". The app inside it is
  dead. This flow probes the resource the way its users reach it, shows the
  terraform plan -replace that would rebuild it, applies that replacement only
  behind auto_remediate plus a human approval, then probes again and fails the
  run when the replacement did not fix it.

labels:
  owner: platform-team
  system: terraform

inputs:
  - id: dry_run
    type: BOOL
    displayName: Dry run
    description: Report the replacement plan without applying it. Safe mode, on by default.
    defaults: true

  - id: auto_remediate
    type: BOOL
    displayName: Auto remediate
    description: Allow the apply. Still needs dry_run false plus an approval on the
      paused execution.
    defaults: false

  - id: docker_host
    type: STRING
    displayName: Docker daemon the provider manages
    description: >-
      Point this at the remote daemon that owns the resource, never at the
      daemon running Kestra. An apply against your own host would churn Kestra
      and every other container on it.
    defaults: tcp://127.0.0.1:2375

  - id: health_url
    type: STRING
    displayName: Health URL
    description: Probed over HTTP from the Terraform container. This is the only
      signal that sees past the container state.
    defaults: http://127.0.0.1:8080/health

  - id: expected_status
    type: INT
    displayName: Expected HTTP status
    description: Any other status counts as degraded. No answer at all counts too.
    defaults: 200

  - id: probe_timeout_seconds
    type: INT
    displayName: Probe timeout (seconds)
    description: Ceiling on the health probe so a hung socket cannot stall the run.
    defaults: 5

  - id: resource_address
    type: STRING
    displayName: Resource address
    description: The address terraform plan -replace and terraform apply -replace target.
    defaults: docker_container.app

  - id: container_name
    type: STRING
    displayName: Container name
    description: Name of the managed container, used by the configuration below.
    defaults: app

  - id: app_image
    type: STRING
    displayName: App image
    description: Image the managed container runs.
    defaults: busybox:1.36

  - id: state_pg_endpoint
    type: STRING
    displayName: State backend host:port
    description: Postgres that holds the Terraform state. The pg backend is built
      into Terraform, no extra provider.
    defaults: postgres:5432

  - id: state_pg_database
    type: STRING
    displayName: State backend database
    description: Database inside that Postgres holding the state table.
    defaults: terraform_state

tasks:
  - id: probe_and_plan
    type: io.kestra.plugin.terraform.cli.TerraformCLI
    description: >-
      Read the resource two ways in one container. Probe the health URL, then
      run terraform plan with a detailed exit code. A clean plan next to a
      failed probe is the finding this flow exists for.
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    containerImage: hashicorp/terraform:1.9
    env:
      PG_CONN_STR: "postgres://{{ secret('POSTGRES_USERNAME') }}:{{
        secret('POSTGRES_PASSWORD') }}@{{ inputs.state_pg_endpoint }}/{{
        inputs.state_pg_database }}?sslmode=disable"
      TF_IN_AUTOMATION: "true"
      TF_VAR_docker_host: "{{ inputs.docker_host }}"
      TF_VAR_container_name: "{{ inputs.container_name }}"
      TF_VAR_app_image: "{{ inputs.app_image }}"
      HEALTH_URL: "{{ inputs.health_url }}"
      PROBE_TIMEOUT: "{{ inputs.probe_timeout_seconds }}"
    inputFiles:
      main.tf: |
        # Replace this with your own configuration. Keep the backend remote so
        # every task in the flow reads and writes the same state.
        terraform {
          required_version = ">= 1.6"
          backend "pg" {}
          required_providers {
            docker = {
              source  = "kreuzwerker/docker"
              version = "~> 3.0"
            }
          }
        }

        variable "docker_host" { type = string }
        variable "container_name" { type = string }
        variable "app_image" { type = string }
        variable "app_port" {
          type    = number
          default = 8080
        }

        provider "docker" {
          host = var.docker_host
        }

        resource "docker_image" "app" {
          name         = var.app_image
          keep_locally = true
        }

        resource "docker_container" "app" {
          name       = var.container_name
          image      = docker_image.app.image_id
          must_run   = true
          restart    = "no"
          entrypoint = ["/bin/sh", "-c"]
          command    = ["mkdir -p /www && printf 'ok' > /www/health && httpd -p ${var.app_port} -h /www && exec sleep 86400"]

          ports {
            internal = var.app_port
            external = var.app_port
          }
        }

        output "container_id" {
          value = docker_container.app.id
        }
    outputFiles:
      - main.tf
      - plan.txt
      - probe.txt
    beforeCommands:
      - terraform init -input=false -no-color
    commands:
      - |
        set +e
        wget -S -T "$PROBE_TIMEOUT" -O probe_body.txt "$HEALTH_URL" > probe.txt 2>&1
        STATUS=$(grep -oE 'HTTP/[0-9.]+ [0-9]{3}' probe.txt | tail -1 | cut -d' ' -f2)
        if [ -z "$STATUS" ]; then STATUS=0; fi
        terraform plan -detailed-exitcode -no-color -input=false > plan.txt 2>&1
        PLAN_EXIT=$?
        CID=$(terraform output -raw container_id 2>/dev/null)
        echo "probe $HEALTH_URL returned HTTP status $STATUS"
        echo "terraform plan detailed exit code $PLAN_EXIT (0 no changes, 2 changes, 1 error)"
        echo "container id recorded in state $CID"
        tail -n 4 plan.txt
        printf '::{"outputs":{"health_status":%s,"plan_exit":%s,"container_id":"%s"}}::\n' "$STATUS" "$PLAN_EXIT" "$CID"
        if [ "$PLAN_EXIT" = "1" ]; then exit 1; fi

  - id: report_finding
    type: io.kestra.plugin.core.log.Log
    description: One line that names the state Terraform reports next to the state
      the resource is actually in.
    message: |
      resource {{ inputs.resource_address }} on {{ inputs.docker_host }}
      health probe {{ inputs.health_url }} returned {{ outputs.probe_and_plan.vars.health_status }}, expected {{ inputs.expected_status }}
      terraform plan exit code {{ outputs.probe_and_plan.vars.plan_exit }} (0 means plan sees nothing to do)
      container id in state {{ outputs.probe_and_plan.vars.container_id }}
      verdict {{ outputs.probe_and_plan.vars.health_status != inputs.expected_status ? 'DEGRADED while plan stays clean. A drift detector would never report this' : 'healthy' }}

  - id: when_degraded
    type: io.kestra.plugin.core.flow.If
    description: Everything below runs only when the probe failed.
    condition: "{{ outputs.probe_and_plan.vars.health_status != inputs.expected_status }}"
    then:
      - id: plan_replacement
        type: io.kestra.plugin.terraform.cli.TerraformCLI
        description: >-
          Read-only. Produce and save the plan that would rebuild the resource,
          so a human reads the exact change before anything is applied.
        taskRunner:
          type: io.kestra.plugin.scripts.runner.docker.Docker
        containerImage: hashicorp/terraform:1.9
        env:
          PG_CONN_STR: "postgres://{{ secret('POSTGRES_USERNAME') }}:{{
            secret('POSTGRES_PASSWORD') }}@{{ inputs.state_pg_endpoint }}/{{
            inputs.state_pg_database }}?sslmode=disable"
          TF_IN_AUTOMATION: "true"
          TF_VAR_docker_host: "{{ inputs.docker_host }}"
          TF_VAR_container_name: "{{ inputs.container_name }}"
          TF_VAR_app_image: "{{ inputs.app_image }}"
        inputFiles:
          main.tf: "{{ outputs.probe_and_plan.outputFiles['main.tf'] }}"
        outputFiles:
          - main.tf
          - replace_plan.txt
          - replace.tfplan
        beforeCommands:
          - terraform init -input=false -no-color
        commands:
          - |
            set +e
            terraform plan -replace={{ inputs.resource_address }} -out=replace.tfplan -no-color -input=false > replace_plan.txt 2>&1
            PLAN_EXIT=$?
            cat replace_plan.txt
            printf '::{"outputs":{"replace_plan_exit":%s}}::\n' "$PLAN_EXIT"
            if [ "$PLAN_EXIT" != "0" ]; then exit 1; fi

      - id: show_replacement_plan
        type: io.kestra.plugin.core.log.Log
        description: Say plainly whether this run stops here.
        message: |
          saved replacement plan for {{ inputs.resource_address }} is ready to review
          dry_run {{ inputs.dry_run }}, auto_remediate {{ inputs.auto_remediate }}
          {{ (inputs.auto_remediate and inputs.dry_run == false) ? 'the run will pause for approval before applying' : 'nothing will be applied on this run' }}

      - id: replacement_gate
        type: io.kestra.plugin.core.flow.If
        description: The write path. Needs auto_remediate true and dry_run false.
        condition: "{{ inputs.auto_remediate and inputs.dry_run == false }}"
        then:
          - id: approve_replacement
            type: io.kestra.plugin.core.flow.Pause
            description: >-
              A replacement destroys the resource before it recreates it, so a
              human answers first. Resume with approved true to apply, false to
              walk away.
            pauseDuration: PT1H
            onResume:
              - id: approved
                type: BOOL
                displayName: Apply the replacement
                description: true applies the saved plan. Anything else leaves the resource
                  alone.
                defaults: false

          - id: when_approved
            type: io.kestra.plugin.core.flow.If
            description: An unanswered or declined pause must change nothing.
            condition: "{{ outputs.approve_replacement.onResume.approved ?? false }}"
            then:
              - id: apply_replacement
                type: io.kestra.plugin.terraform.cli.TerraformCLI
                description: Apply the exact plan the human reviewed. This destroys and
                  recreates the resource.
                taskRunner:
                  type: io.kestra.plugin.scripts.runner.docker.Docker
                containerImage: hashicorp/terraform:1.9
                env:
                  PG_CONN_STR: "postgres://{{ secret('POSTGRES_USERNAME') }}:{{
                    secret('POSTGRES_PASSWORD') }}@{{ inputs.state_pg_endpoint
                    }}/{{ inputs.state_pg_database }}?sslmode=disable"
                  TF_IN_AUTOMATION: "true"
                  TF_VAR_docker_host: "{{ inputs.docker_host }}"
                  TF_VAR_container_name: "{{ inputs.container_name }}"
                  TF_VAR_app_image: "{{ inputs.app_image }}"
                inputFiles:
                  main.tf: "{{ outputs.plan_replacement.outputFiles['main.tf'] }}"
                  replace.tfplan: "{{ outputs.plan_replacement.outputFiles['replace.tfplan'] }}"
                outputFiles:
                  - main.tf
                  - apply.txt
                beforeCommands:
                  - terraform init -input=false -no-color
                commands:
                  - |
                    set +e
                    terraform apply -no-color -input=false replace.tfplan > apply.txt 2>&1
                    APPLY_EXIT=$?
                    cat apply.txt
                    NEW_CID=$(terraform output -raw container_id 2>/dev/null)
                    echo "container id after the replacement $NEW_CID"
                    printf '::{"outputs":{"apply_exit":%s,"container_id":"%s"}}::\n' "$APPLY_EXIT" "$NEW_CID"
                    if [ "$APPLY_EXIT" != "0" ]; then exit 1; fi

              - id: reverify
                type: io.kestra.plugin.terraform.cli.TerraformCLI
                description: >-
                  Read the system again. The apply going green proves nothing,
                  so probe the health URL once more and re-run plan.
                taskRunner:
                  type: io.kestra.plugin.scripts.runner.docker.Docker
                containerImage: hashicorp/terraform:1.9
                env:
                  PG_CONN_STR: "postgres://{{ secret('POSTGRES_USERNAME') }}:{{
                    secret('POSTGRES_PASSWORD') }}@{{ inputs.state_pg_endpoint
                    }}/{{ inputs.state_pg_database }}?sslmode=disable"
                  TF_IN_AUTOMATION: "true"
                  TF_VAR_docker_host: "{{ inputs.docker_host }}"
                  TF_VAR_container_name: "{{ inputs.container_name }}"
                  TF_VAR_app_image: "{{ inputs.app_image }}"
                  HEALTH_URL: "{{ inputs.health_url }}"
                  PROBE_TIMEOUT: "{{ inputs.probe_timeout_seconds }}"
                inputFiles:
                  main.tf: "{{ outputs.apply_replacement.outputFiles['main.tf'] }}"
                outputFiles:
                  - reprobe.txt
                  - replan.txt
                beforeCommands:
                  - terraform init -input=false -no-color
                commands:
                  - |
                    set +e
                    sleep 3
                    wget -S -T "$PROBE_TIMEOUT" -O reprobe_body.txt "$HEALTH_URL" > reprobe.txt 2>&1
                    STATUS=$(grep -oE 'HTTP/[0-9.]+ [0-9]{3}' reprobe.txt | tail -1 | cut -d' ' -f2)
                    if [ -z "$STATUS" ]; then STATUS=0; fi
                    terraform plan -detailed-exitcode -no-color -input=false > replan.txt 2>&1
                    PLAN_EXIT=$?
                    CID=$(terraform output -raw container_id 2>/dev/null)
                    echo "re-probe $HEALTH_URL returned HTTP status $STATUS"
                    echo "terraform plan detailed exit code $PLAN_EXIT after the replacement"
                    echo "container id now in state $CID"
                    printf '::{"outputs":{"health_status":%s,"plan_exit":%s,"container_id":"%s"}}::\n' "$STATUS" "$PLAN_EXIT" "$CID"

              - id: fail_when_still_degraded
                type: io.kestra.plugin.core.execution.Fail
                description: The replacement is only done when the probe passes. Fail the run
                  otherwise.
                condition: "{{ outputs.reverify.vars.health_status != inputs.expected_status }}"
                errorMessage: >-
                  Replaced {{ inputs.resource_address }} but {{
                  inputs.health_url }} still answers {{
                  outputs.reverify.vars.health_status }} instead of {{
                  inputs.expected_status }}. The resource is new, the fault is
                  not in the resource.

              - id: replacement_confirmed
                type: io.kestra.plugin.core.log.Log
                description: Before and after, read back from the system itself.
                message: |
                  replaced {{ inputs.resource_address }}
                  container id before {{ outputs.probe_and_plan.vars.container_id }}
                  container id after {{ outputs.reverify.vars.container_id }}
                  health probe now {{ outputs.reverify.vars.health_status }}
                  terraform plan exit code now {{ outputs.reverify.vars.plan_exit }}

            else:
              - id: replacement_declined
                type: io.kestra.plugin.core.log.Log
                description: Nothing was applied. Say so.
                message: |
                  replacement of {{ inputs.resource_address }} was declined or the pause went unanswered
                  the saved plan stays on the execution for the next run
                  nothing was destroyed, nothing was recreated

    else:
      - id: no_findings
        type: io.kestra.plugin.core.log.Log
        description: Healthy resource, clean plan, no work to do.
        message: |
          {{ inputs.resource_address }} answered {{ outputs.probe_and_plan.vars.health_status }} on {{ inputs.health_url }}
          terraform plan exit code {{ outputs.probe_and_plan.vars.plan_exit }}
          nothing to replace

errors:
  - id: notify_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Tell the on-call channel which stage failed.
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": ":rotating_light: terraform-degraded-resource-replace-guard failed on {{ inputs.resource_address }} ({{ inputs.docker_host }}). Execution {{ execution.id }}, flow {{ flow.namespace }}.{{ flow.id }}."
      }

triggers:
  - id: hourly_degraded_check
    type: io.kestra.plugin.core.trigger.Schedule
    description: Hourly probe in report-only mode. Enable it once the health URL and
      the resource address are yours.
    cron: "17 * * * *"
    disabled: true
    inputs:
      dry_run: true
      auto_remediate: false

Findings

The flow emits one verdict per run: the HTTP status the health probe got, the terraform plan detailed exit code, the container id recorded in state and, after a replacement, the new container id. A degraded verdict next to a plan exit code of 0 is the case no drift detector reports.

Read them as {{ outputs.probe_and_plan.vars.health_status }}, {{ outputs.probe_and_plan.vars.plan_exit }} and {{ outputs.probe_and_plan.vars.container_id }}, with the same three names on outputs.reverify.vars after a replacement. The raw text stays on the execution as files: {{ outputs.probe_and_plan.outputFiles['plan.txt'] }}, {{ outputs.plan_replacement.outputFiles['replace_plan.txt'] }} and the binary plan at {{ outputs.plan_replacement.outputFiles['replace.tfplan'] }}.

Safety

Read-only by default. dry_run is true and auto_remediate is false, so a plain run probes the resource, saves the replacement plan and stops. The apply needs auto_remediate true, dry_run false and a human answering approved: true on the paused execution. An unanswered pause changes nothing.

Point docker_host at the remote daemon that owns the resource. Never at the daemon running Kestra. terraform apply -replace destroys before it recreates, so a provider aimed at your own host would churn Kestra and every other container beside it.

How it works

A resource can match its configuration byte for byte and still be useless. A container is RUNNING, so the Docker provider reads back the same name, the same image and the same ports the state file records. terraform plan prints "No changes. Your infrastructure matches the configuration." The process inside it died twenty minutes ago.

probe_and_plan runs one Terraform container that does both reads. It fetches the health URL over HTTP, then runs terraform plan -detailed-exitcode. The probe status and the plan exit code come back as task outputs.
report_finding logs both numbers side by side, so the paradox is in the run log rather than in somebody's head.
plan_replacement runs only when the probe failed. It produces terraform plan -replace=<address> and saves the binary plan, so the change a human reviews is the change that gets applied.
approve_replacement pauses. apply_replacement applies the saved plan behind that approval.
reverify probes the health URL again and re-runs plan. fail_when_still_degraded fails the run when the probe is still bad, because a rebuilt resource that is still broken means the fault was never in the resource.

The Terraform project is declared once in probe_and_plan then travels to the later tasks as a task output file. State lives in Postgres through Terraform's built-in pg backend, so every task in the flow reads and writes the same state and the saved plan stays valid across the pause.

What you get

A detector for the failure mode drift detection is blind to: config matches state, resource is wedged.
The replacement shown as a saved plan before anything is destroyed.
terraform apply -replace run against a reviewed plan, never a fresh one.
A re-probe that fails the run when the rebuild did not help.
A Slack alert naming the resource and the daemon when a stage fails.

Who it's for

Platform and SRE teams who manage long-lived resources with Terraform and have been paged for something plan swore was fine. Anyone running containers, VMs or managed instances where liveness is not visible in the provider's read.

Why orchestrate this with Kestra

The pieces are a probe, a plan, an approval and a second probe, with a destructive step in the middle. A shell script can run the commands. It cannot hold a plan file across a one hour human approval. It cannot keep the state backend credentials off the command line or record both reads as outputs an operator can inspect weeks later. It cannot fail the run on the re-probe instead of on the apply. Kestra gives the pause, the typed inputs that keep the default read-only, the task outputs and the failure branch.

Tested end to end

Run on Kestra 2.0.5 with plugin-terraform 1.2.10, Terraform 1.9.8 and the kreuzwerker/docker provider 3.9.0 against a Docker-in-Docker daemon and a Postgres state backend.

Healthy resource: probe 200, plan exit 0, zero findings, nothing planned.
Wedged resource: the app process killed inside a container left RUNNING. Probe 0 (connection reset), plan exit 0 with "No changes. Your infrastructure matches the configuration." The detector fired on the same inputs that returned nothing a minute earlier.
dry_run: true: the replacement plan was saved and logged (1 to add, 1 to destroy), nothing applied, container id unchanged.
auto_remediate: true, dry_run: false, approved: the container was replaced and the re-probe returned 200.
Pause declined: no apply, container id unchanged.
Re-verify failure: a health URL the replacement cannot satisfy failed the run on the re-probe, not on the apply.
Failure path: a wrong state backend endpoint failed terraform init and the errors block posted to Slack.

Prerequisites

Secrets

POSTGRES_USERNAME, POSTGRES_PASSWORD: the Terraform state backend.
SLACK_WEBHOOK_URL: incoming webhook for the failure alert.

Inputs

dry_run (BOOL, default true): report only.
auto_remediate (BOOL, default false): allow the apply.
docker_host: the remote Docker daemon the provider manages.
health_url, expected_status, probe_timeout_seconds: the probe.
resource_address: what -replace targets.
container_name, app_image: the managed resource.
state_pg_endpoint, state_pg_database: the pg state backend.

Quick start

Add the three secrets.
Replace the main.tf in probe_and_plan with your own configuration and backend. Keep an output that exposes the resource id so the before and after values show up in the log.
Set docker_host to the remote daemon, health_url to something your users actually hit, resource_address to the address you would type after -replace.
Run it with the defaults. You get a verdict, a saved replacement plan and no changes.
To remediate, run with auto_remediate: true and dry_run: false, then resume the paused execution with approved: true.
Enable the Schedule trigger once the report-only runs look right.

How to extend

Swap the provider. Any resource whose provider read cannot see liveness fits: an EC2 instance that passes its system check with a dead service, an ECS task, a Cloud SQL replica that stopped replicating.
Probe something sharper than a health URL. A query that must return rows, a queue depth, a replication lag.
Loop over several addresses with core.flow.Loop and concurrencyLimit: 1, so two replacements never race on one state lock.
Replace the Slack task with PagerDuty or Opsgenie for the on-call path.
Drop plan_replacement and apply -replace directly when you trust the detector enough to skip the review step.

Replace a Terraform resource that plan says is fine

Findings

Safety

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Replace a Terraform resource that plan says is fine

Findings

Safety

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Tested end to end

Prerequisites

Secrets

Inputs

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Purge Fastly Cache on Deploy via Webhook

Fastly Emergency Purge All with Confirmation Gate

Faktory Job Enqueue Path Reachability Sentinel

Fastly Edge Error Spike Alert with the Stats Trigger

Fastly Daily Traffic Digest to Slack

Monitor FastAPI Synthetic Latency SLOs and Alert SRE Teams via Slack

New to Kestra?