Search icon
Get icon
OutputValues icon
Switch icon
Log icon
SlackIncomingWebhook icon
Fail icon
Create icon
Write icon
Schedule icon
Webhook icon

Quickwit Selective Deletion and Retention Blast Radius Gate

Audit and gate Quickwit document deletions by evaluating matched record blast radius before queuing irreversible janitor purge tasks on object storage.

Categories
DataInfrastructure

Automate GDPR Data Subject Access Request (DSAR) erasures, tenant offboarding expulsions, and cold log retention pruning across Quickwit search clusters with rigorous pre-flight safety gates and blast radius guardrails.

Quickwit decouples compute from storage by indexing terabytes of logs, events, and distributed traces directly into immutable split files hosted on cloud object storage (Amazon S3, Google Cloud Storage, or MinIO). In Quickwit, deletions are executed asynchronously through delete tasks (deletetask.Create): the task registers a query in the cluster metastore, and the background janitor rewrites affected splits, permanently removing matched documents.

Because janitor deletions are irreversible and rewrite underlying object storage splits, an unvetted query containing syntax errors, missing field prefixes, or accidental wildcards (*) can inadvertently purge an entire index. This blueprint establishes a multi-tiered operational gate before any delete task is queued: it runs a pre-flight count probe, validates cluster metadata, evaluates the matched record count against strict blast radius ceilings, enforces a mandatory dry-run simulation mode, and generates an immutable JSON compliance certificate in Kestra internal storage upon execution.

How it works

  1. Pre-Flight Count & Sample Probe (preflight_search): Executes an immediate search query against the target Quickwit index using io.kestra.plugin.quickwit.search.Search. It fetches up to 5 representative document samples and extracts the exact total matching record count (outputs.preflight_search.total) without modifying index state.
  2. Index Metadata Verification (get_index_metadata): Queries io.kestra.plugin.quickwit.index.Get to confirm cluster reachability, index existence, and active configuration before evaluating risk.
  3. Operational Risk Tier Classification (classify_risk_tier): Dynamically computes risk tier via io.kestra.plugin.core.output.OutputValues:
    • ZERO_MATCH: Total matching records equal 0. No deletion task is needed.
    • UNSAFE_BLAST_RADIUS: Triggered if the query matches all records (*), or if matched records exceed max_allowed_deletion_count.
    • DRY_RUN_PREVIEW: Triggered when records match within bounds but dry_run: true or auto_remediate: false.
    • PROCEED_DELETION: Triggered only when records are within safe thresholds and operator explicitly authorizes execution with dry_run: false and auto_remediate: true.
  4. Safety Branching (route_by_risk_tier): Evaluates the risk tier using io.kestra.plugin.core.flow.Switch:
    • Under ZERO_MATCH, logs an audit entry confirming compliance without creating cluster load.
    • Under UNSAFE_BLAST_RADIUS, posts a high-priority security alert to Slack via io.kestra.plugin.slack.notifications.SlackIncomingWebhook and fails the execution via io.kestra.plugin.core.execution.Fail to quarantine the operation.
    • Under DRY_RUN_PREVIEW, logs matched records and delivers a preview summary to Slack detailing matched document counts and search latency.
    • Under PROCEED_DELETION, registers the deletion task in Quickwit via io.kestra.plugin.quickwit.deletetask.Create, writes a tamper-evident compliance audit certificate to Kestra internal storage using io.kestra.plugin.core.storage.Write, and sends a Slack notification with the cluster opstamp.
  5. Audit Logging & Triggers: Emits an end-of-run log entry with execution ID and query parameters. Supports scheduled weekly maintenance via io.kestra.plugin.core.trigger.Schedule and on-demand DSAR requests via io.kestra.plugin.core.trigger.Webhook.

What you get

  • Fail-safe pre-flight dry-run evaluation that blocks accidental wildcard deletions (* and *:*).
  • Configurable max_allowed_deletion_count blast radius ceiling preventing runaway data loss.
  • Automated risk tier classification separating zero-match, dry-run, unsafe, and authorized states.
  • Asynchronous delete task registration using native Quickwit plugin classes (deletetask.Create).
  • Immutable JSON compliance certificates saved to Kestra internal storage with execution ID, opstamp, and timestamp.
  • Operational Slack notifications for critical blast radius breaches, dry-run previews, and successful purge queueing.

Who it's for

  • Platform Engineers and SREs administering Quickwit search and observability clusters on object storage.
  • Data Governance Officers and Compliance Engineers managing automated GDPR DSAR right-to-be-forgotten requests.
  • SecOps Teams enforcing bounded retention policies across multi-tenant log aggregation architectures.

Why orchestrate this with Kestra

Quickwit's delete task API only registers deletion requests in the metastore; it provides no pre-flight safety checking, dry-run estimation, or blast radius gating. Hand-rolled curl scripts run the risk of passing unescaped queries that delete entire indexes, leaving no auditable execution record. Kestra wraps the lifecycle in declarative YAML: validating index existence, verifying document counts before dispatch, isolating credentials in encrypted secrets, enforcing human dry-run authorization, and generating durable audit certificates.

Prerequisites

  • A running Quickwit cluster reachable from the Kestra worker, accessible via HTTP/HTTPS.
  • A Quickwit index (e.g. app-logs or audit-events) with searchable timestamp and metadata fields.
  • A Slack Incoming Webhook endpoint URL configured in Kestra namespace secrets.

Secrets

  • QUICKWIT_URL: Base HTTP or HTTPS URL of the Quickwit cluster (e.g., http://quickwit.internal:7280).
  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used for blast radius alerts, dry-run previews, and completion notices.
  • QUICKWIT_GATE_WEBHOOK_KEY: Shared secret key used to authenticate inbound HTTP webhook execution requests.

Quick start

  1. Configure secrets QUICKWIT_URL, SLACK_WEBHOOK_URL, and QUICKWIT_GATE_WEBHOOK_KEY in your Kestra namespace.
  2. Import this blueprint into your Kestra instance under namespace company.governance.
  3. Run a test execution with default inputs (dry_run: true) to preview matched records without modifying the Quickwit index.
  4. Confirm matched document count and latency in execution logs or Slack preview card.
  5. When ready to delete, execute with dry_run: false, auto_remediate: true, and your specific tenant or timestamp query.

How to extend

  • Janitor Split Convergence Check: Chain an io.kestra.plugin.quickwit.deletetask.List polling loop to monitor until the queued opstamp has been compacted across all splits.
  • Cold Archive Offload: Before queuing deletion, export matching documents to cold compressed Parquet on AWS S3 or MinIO using io.kestra.plugin.quickwit.search.Search with fetchType: STORE.
  • Multi-Tenant Fan-Out: Use a io.kestra.plugin.core.flow.Loop task to iterate through a list of offboarded tenant IDs from an external PostgreSQL or Snowflake database.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.