Schedule icon
WorkingDirectory icon
TerraformCLI icon
Docker icon
Script icon
If icon
SlackIncomingWebhook icon
Fail icon
Log icon

Terraform Drift Detection & Remediation Planner

Detect out-of-band Terraform drift using refresh-only planning, assess operational risk, alert Slack, and generate advisory remediation runbooks with Kestra.

Categories
Infrastructure

Cloud infrastructure drifts. Emergency console edits, direct API calls, manual security group tweaks, and automated platform updates slowly detach live cloud environments from version-controlled Terraform state files. Unmanaged drift creates invisible vulnerabilities, breaks subsequent deployments, and causes catastrophic surprises when routine updates inadvertently destroy undocumented out-of-band resources.

While standard CI/CD pipelines run terraform plan on pull requests, regular planning only checks code changes awaiting deployment, it completely misses drift accumulating quietly in production. Conversely, ad-hoc terraform apply commands can silently overwrite emergency fixes or trigger unexpected resource replacements.

This blueprint implements a safe, automated Terraform Drift Detection & Remediation Planner. Using Terraform's dedicated -refresh-only planning mode, it queries live cloud APIs to detect divergences against recorded state without proposing or applying infrastructure mutations. A Python risk engine introspects the plan JSON, classifies operational blast radius, redacts sensitive attributes, alerts team channels via Slack, and produces structured remediation runbooks for human operators.

Architecture

Schedule Trigger (Daily 06:00 UTC) / Manual Execution
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│ WorkingDirectory (drift_workspace)                          │
│                                                             │
│  1. TerraformCLI (detect_drift)                             │
│     - Image: hashicorp/terraform:1.9.8                      │
│     - terraform init -input=false                           │
│     - terraform plan -refresh-only -detailed-exitcode       │
│     - terraform show -json tfdrift > drift_plan.json        │
│                                                             │
│  2. Python Script (analyze_and_plan_remediation)            │
│     - Image: python:3.11-slim                               │
│     - Isolates resource_drift block from regular changes    │
│     - Evaluates risk: CRITICAL, HIGH, MEDIUM, LOW           │
│     - Redacts sensitive attributes in logs and summaries   │
│     - Emits drift_report.json & drift_summary.md            │
└──────────────────────────────┬──────────────────────────────┘
                               │
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Flow Control: If (evaluate_drift_results)                   │
│ Condition: outputs.analyze_and_plan_remediation.drift_detected│
│                                                             │
│  Then:                                                      │
│   ├── If [notify_slack]: Slack Webhook Notification         │
│   └── If [fail_on_critical_drift]: Fail Task (Halt Gate)    │
│                                                             │
│  Else:                                                      │
│   └── Log: Clean state confirmation                         │
└─────────────────────────────────────────────────────────────┘

How it works

  1. Triggering: Fires on a configured cron schedule (0 6 * * *, daily 06:00 UTC, shipped disabled by default) or executes manually on demand via the Kestra UI or REST API.
  2. Isolated Workspace: io.kestra.plugin.core.flow.WorkingDirectory sets up an execution directory with the Terraform configuration and captures output artifacts (drift_report.json, drift_summary.md, and drift_plan.json).
  3. Safe Refresh-Only Planning: io.kestra.plugin.terraform.cli.TerraformCLI runs terraform init, followed by terraform plan -refresh-only -detailed-exitcode -out=tfdrift. The command evaluates live cloud provider APIs and updates state in memory without executing any infrastructure changes. The binary plan is then exported to JSON via terraform show -json tfdrift > drift_plan.json.
  4. Plan Introspection & Risk Classification: An embedded Python analysis engine (io.kestra.plugin.scripts.python.Script on python:3.11-slim) parses the JSON plan:
    • Drift Isolation: Distinguishes out-of-band resource_drift from regular plan resource_changes.
    • Sensitive Data Redaction: Inspects before_sensitive and after_sensitive properties and masks secret values with (sensitive value redacted) across all reports and notifications.
    • Heuristic Risk Scoring: Categorizes findings into CRITICAL, HIGH, MEDIUM, or LOW based on resource types (stateful databases, object stores, network boundaries, security groups) and modified attributes.
    • Artifact Generation: Writes structured drift_report.json and human-readable drift_summary.md.
  5. Flow Control & Alerting:
    • io.kestra.plugin.core.flow.If evaluates outputs.analyze_and_plan_remediation.vars.drift_detected.
    • If drift is detected and notify_slack: true is enabled, io.kestra.plugin.slack.notifications.SlackIncomingWebhook delivers a structured alert including overall risk level, drifted resource counts, advisory remediation recommendations, and execution links.
    • If fail_on_critical_drift: true is configured and risk is evaluated as CRITICAL, io.kestra.plugin.core.execution.Fail halts execution to enforce change control.
    • If no drift is detected, a clean verification message is logged.
  6. Error Boundary: A dedicated errors task logs unexpected execution failures (such as provider connectivity or authentication issues), guaranteeing that a broken scan never masquerades as a clean result.

What you get

  • Scheduled Out-of-Band Visibility: Automated continuous audit of cloud infrastructure without standing up complex third-party drift detection SaaS tools.
  • Zero Mutation Guarantee: Executes strictly read-only refresh-only plans; never calls terraform apply or mutates infrastructure.
  • Blast-Radius Classification: Instant categorization distinguishing harmless tag drift from dangerous security group widening or stateful storage deletion.
  • Sensitive Value Masking: Automated scrubbing of passwords, tokens, and secret attributes marked sensitive in provider schemas.
  • Actionable Advisory Runbooks: Markdown and JSON runbooks detailing step-by-step guidance on accepting intentional drift (terraform apply -refresh-only) versus reverting unauthorized modifications.
  • Dual Operational Modes: Supports non-blocking advisory observation (default) as well as enforcing pipeline gating.

Who it's for

  • DevOps & Platform Engineers: Maintain consistency between Git-managed Terraform configurations and production cloud estates.
  • Site Reliability Engineers (SREs): Early warning detection of out-of-band infrastructure changes before they trigger operational incidents.
  • Security & Compliance Teams: Audit unauthorized modifications to IAM policies, security groups, and encryption configurations.

Why orchestrate this with Kestra

Terraform CLI has no native scheduling, alerting, or execution history. Running drift checks in traditional CI/CD engines requires custom shell glue, fragile artifact storage, and separate webhook handlers.

Kestra coordinates the entire drift management lifecycle:

  • Declarative Workspace Isolation: Mounts configurations, executes containers, and retains generated JSON and Markdown artifacts for audit compliance.
  • Dynamic Flow Control: Evaluates structured outputs from plan introspection scripts to branch conditionally into Slack alerts, tickets, or failure gates.
  • Built-in Credential Management: Injects cloud credentials and webhook URLs securely via Kestra secrets ({{ secret(...) }}) without hardcoding credentials in repository files.
  • Audit History: Maintains complete execution lineage, logs, and downloadable reports for compliance audits.

Risk Heuristics & Severity Classification

Risk Level Trigger Criteria Example Scenarios
CRITICAL Out-of-band deletion of stateful or security resources; widening of network security perimeters. Production RDS database deleted outside Terraform; security group opened to 0.0.0.0/0; S3 bucket deleted.
HIGH Out-of-band modification of security groups, IAM policies, KMS keys, or deletion of non-stateful resources. Firewall rule modified via console; IAM role policy altered; compute instance terminated out-of-band.
MEDIUM In-place operational configuration drift on standard compute or application resources. VM machine type altered; timeout or memory limit modified; scaling parameter changed.
LOW Non-functional metadata drift with zero operational or security impact. Resource tags, labels, or description fields updated out-of-band.
NONE No divergence between Terraform state and live infrastructure. All resources in state perfectly match cloud reality.

Note: Risk classifications are heuristic assessments based on provider resource schemas and attribute deltas. Ambiguous or unknown resource types are evaluated conservatively as HIGH risk for human review.

Advisory Remediation Playbook

When drift is detected, this blueprint provides three advisory decision paths for operators:

  1. Accept Intentional Drift (Cloud is Right): If the live cloud changes were intentional (e.g. authorized hotfixes or platform updates), synchronize state without altering live resources:
    terraform apply -refresh-only
    
    Then update the version-controlled HCL configuration in Git to reflect the new state, commit, and push.
  2. Revert Unauthorized Drift (Code is Right): If the cloud changes were unauthorized or accidental, revert live infrastructure back to match your version-controlled code:
    terraform plan
    terraform apply
    
  3. Manual Triage (Critical Discrepancies): For stateful resource deletions or security breaches, do not automate apply. Inspect cloud audit trails (AWS CloudTrail, GCP Cloud Audit Logs, Azure Activity Log) to identify root cause before reconciling state.

Security & Safe Operations

  • Zero Execution Policy: This blueprint never executes terraform apply, terraform destroy, or state removal commands.
  • State Locking Preserved: Runs with native state locking enabled; state locks are never bypassed.
  • Zero Plaintext Secrets: Secrets such as SLACK_WEBHOOK_URL, AWS_ACCESS_KEY_ID, or AWS_SECRET_ACCESS_KEY are referenced strictly via Kestra's {{ secret(...) }} store.
  • Fail-Closed Architecture: If Terraform fails to initialize or the plan cannot be exported, the workflow terminates with an explicit error rather than reporting a false clean status.

Prerequisites & Secrets

  • Container Engine: Docker or container runner accessible to Kestra for executing hashicorp/terraform:1.9.8 and python:3.11-slim.
  • SLACK_WEBHOOK_URL (optional): Incoming Slack webhook URL for alerts. Required only when notify_slack: true.
  • Cloud Provider Credentials (optional): For cloud deployments (AWS, GCP, Azure), configure credentials as environment variables or Kestra namespace secrets matching your provider configuration.

Quick Start

  1. Import this blueprint into your Kestra instance.
  2. Run the workflow manually with default inputs: it executes against the embedded safe local/random configuration and confirms zero drift.
  3. Enable Slack alerts by setting notify_slack: true and defining the SLACK_WEBHOOK_URL secret.
  4. To monitor your real cloud estate, replace the default terraform_code input with your HCL configuration and remote backend block (or mount via Git as shown below).
  5. Enable the daily_drift_schedule trigger by setting disabled: false.

How to Extend

  • Git Clone Integration: Replace the inline terraform_code input with io.kestra.plugin.git.Clone inside WorkingDirectory to pull your IaC repository directly from GitHub, GitLab, or Bitbucket.
  • Issue Tracker Automation: Chain io.kestra.plugin.github.issues.Create or Jira tasks inside the If block to automatically open tracking tickets for high-risk drift findings.
  • PagerDuty Escalation: Trigger PagerDuty incidents when outputs.analyze_and_plan_remediation.vars.risk_level == 'CRITICAL'.
  • Multi-Cloud Credentials: Add provider environment variables (such as GOOGLE_APPLICATION_CREDENTIALS or ARM_CLIENT_SECRET) to the TerraformCLI task.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.