DuckDB Parquet Lakehouse Schema Evolution Guard

In modern cloud data lakehouses (built on S3, Google Cloud Storage, or Azure Blob Storage), upstream services continuously land daily or hourly Parquet partitions. Unannounced schema changes made by application developers—such as renaming a column, altering an integer field to a string, or dropping a required dimension—silently break downstream analytics engines, BI dashboards, and dbt models.

Because Parquet embeds metadata in each file's footer, inspecting schema drift without loading multi-gigabyte datasets into memory is challenging.

This blueprint implements an automated lakehouse schema contract gate using in-memory DuckDB. Executing daily or upon partition landing, it calls the native parquet_schema() table function across target and baseline partitions, performs a full outer join on column metadata, and categorizes differences as DROPPED_COLUMN, NEW_COLUMN, or DATA_TYPE_MUTATION. If breaking changes are found, it dispatches an immediate alert to Slack.

How it works

1. Scheduled Schema Audit: The daily_schema_audit trigger (io.kestra.plugin.core.trigger.Schedule) executes daily at 07:00 UTC or on-demand via the Kestra UI.
2. DuckDB Schema Diff: The audit_parquet_schema_drift task (io.kestra.plugin.jdbc.duckdb.Query) loads the parquet and httpfs extensions, reads file footers via parquet_schema(), and joins column metadata to classify structural changes.
3. Evaluation Gate: The evaluate_schema_breaking_changes flowable task (io.kestra.plugin.core.flow.If) evaluates whether any breaking column deviations exist.
4. Slack Alert Dispatch: If schema drift is detected, notify_data_engineering_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts an alert with column names, expected vs. actual types, and remediation guidance.
5. Compliant Confirmation: When file schemas match the baseline contract, log_compliant_schema records verification in execution logs.
6. Audit Manifest: The export_schema_manifest task records contract verification metadata for pipeline lineage records.

What you get

• Zero-copy Parquet schema inspection reading only file footers rather than full datasets.
• Automated detection of dropped columns and breaking data type mutations.
• Configurable sensitivity for additive new columns via fail_on_new_columns.
• Immediate Slack notifications alerting data engineers before downstream transformations fail.

Who it is for

• Data Platform Engineers maintaining S3/GCS data lakehouses and Iceberg/Delta Lake tables.
• Analytics Engineers debugging recurring schema mismatch errors in dbt pipelines.
• Data Governance teams enforcing upstream producer data contracts.

Why orchestrate this with Kestra

Validating Parquet schemas across cloud object storage typically requires heavy Spark jobs or custom Python scripts. Kestra combines serverless workflow orchestration with DuckDB's in-process analytical power: it inspects remote Parquet files over HTTP/S3, executes fast metadata queries, branches conditionally, and dispatches structured alerts with minimal compute overhead.

Inputs

Name	Type	Default	Description
target_parquet_uri	STRING	https://huggingface.co/datasets/...	URI or path to the freshly landed daily Parquet partition.
baseline_parquet_uri	STRING	https://huggingface.co/datasets/...	URI or path to the validated baseline Parquet file.
fail_on_new_columns	BOOLEAN	false	If false, new columns are permitted; if true, any additive column triggers an alert.
slack_channel	STRING	#data-governance	Slack channel destination for schema drift alerts.

Expected outputs

• {{ outputs.audit_parquet_schema_drift.rows }}: Array of drifting column records containing column_name, expected_type, actual_type, and drift_status.
• {{ outputs.audit_parquet_schema_drift.size }}: Total number of schema breaking changes detected.
• {{ outputs.evaluate_schema_breaking_changes }}: Result of conditional branch evaluation.
• {{ outputs.export_schema_manifest.value }}: Structured JSON audit manifest recording verification timestamp and contract status.

Prerequisites

• Accessibility to target and baseline Parquet files (public HTTP URL or private S3/GCS bucket).
• If reading private cloud buckets, configure cloud credentials (e.g. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) in DuckDB secrets.
• A Slack Incoming Webhook URL targeting your data governance channel.

Secrets

• SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for sending alert notifications.

Quick start

1. Configure SLACK_WEBHOOK_URL in your Kestra namespace secrets.
2. Import this flow YAML into your Kestra instance.
3. Click Execute in the UI to perform an initial schema diff between sample Parquet files.
4. Review execution outputs to inspect column type comparisons and drift classifications.

Common pitfalls and troubleshooting

• HTTPFS Extension Installation: DuckDB requires the httpfs extension to read remote URLs over HTTPS or S3. The query automatically includes INSTALL httpfs; LOAD httpfs;.
• Physical vs Logical Types: Parquet supports physical and logical types (e.g., INT64 with logical timestamp). DuckDB exposes the unified SQL type representation for clean comparisons.
• S3 Permissions: When auditing private S3 buckets, add SET s3_region = '...'; SET s3_access_key_id = '...'; commands to the query preamble.

How to extend

• Add io.kestra.plugin.aws.s3.List to automatically discover the latest partition file in an S3 prefix and compare it against the preceding partition.
• Integrate with data catalog registries to automatically update table schema definitions when non-breaking additive columns are discovered.
• Halt downstream dbt or Spark ingestion jobs when breaking schema mutations are flagged.

Links

• Kestra DuckDB Plugin Documentation: https://kestra.io/plugins/plugin-jdbc-duckdb
• Kestra Slack Plugin Documentation: https://kestra.io/plugins/plugin-notifications
• DuckDB Parquet Metadata Documentation: https://duckdb.org/docs/data/parquet/metadata