Query icon
Queries icon
Switch icon
Log icon
Fail icon

DuckDB Parquet Dictionary Encoding Fallback Guard

Check Parquet column chunk metadata with DuckDB for categorical columns that fell back from dictionary to PLAIN encoding, and fail when too many degrade.

Categories
Data

Checks a Parquet file for silent loss of dictionary encoding before it is loaded into a lakehouse or warehouse. Categorical columns such as status codes, country codes and payment methods normally compress through dictionary encoding. When a producer writes high-cardinality values into one of them (random tokens, UUIDs, unparsed timestamps), the writer's dictionary page overflows and the column falls back to PLAIN encoding. File size grows, filter pushdown slows down, and no row count or file size check shows it.

The flow reads parquet_metadata() with embedded DuckDB, groups the column chunks by column across all row groups, and reports every column you listed as dictionary encoded in all row groups, partly or not at all. It writes a per-column CSV and fails the execution when the number of degraded columns is above your limit, or when a listed column does not exist in the file. It only reads metadata and never modifies the file.

How it works

  1. profile_dictionary_encoding (io.kestra.plugin.jdbc.duckdb.Query, fetchType: FETCH_ONE) mounts parquet_uri with inputFiles and aggregates parquet_metadata() per column. A column counts as healthy when every row group has a dictionary encoding in its encodings, and as degraded when at least one row group does not. Columns you listed that are missing from the file are counted separately, so a typo cannot pass silently.
  2. The query returns one verdict: HEALTHY, DEGRADED_WITHIN_TOLERANCE, CRITICAL_DICTIONARY_OVERFLOW or COLUMNS_NOT_FOUND.
  3. write_encoding_report (io.kestra.plugin.jdbc.duckdb.Queries) writes dictionary_encoding_report.csv with, for each listed column, the row group count, the number of dictionary-encoded row groups, uncompressed and compressed bytes, the compression ratio and a status.
  4. route_by_encoding_health (io.kestra.plugin.core.flow.Switch) acts on the verdict:
    • HEALTHY: logs the result.
    • DEGRADED_WITHIN_TOLERANCE: logs a warning that names the degraded columns.
    • CRITICAL_DICTIONARY_OVERFLOW: fail_on_dictionary_overflow (Fail) fails the execution.
    • COLUMNS_NOT_FOUND: fail_on_missing_columns (Fail) fails the execution.

What you get

  • A check of Parquet encoding that file size and row count checks cannot give you.
  • Per-column results across all row groups, so a column that degrades only in later row groups is still caught.
  • A CSV report with compression ratios for the columns you care about.
  • A failed execution, rather than a pass, when a listed column is absent.

Who it's for

  • Data Lakehouse Engineers: Teams that operate Parquet, Iceberg or Delta storage.
  • Analytics Engineers: Teams that watch query latency and storage cost on warehouses and DuckDB.
  • Data Platform Engineers: Engineers who add quality gates to batch landing pipelines.

Why orchestrate this with Kestra

The check is cheap but easy to forget. In Kestra it runs right after a file lands, leaves a per-column report as an execution output and fails the execution so downstream flows can stop. A Switch keeps the policy in one visible place.

Prerequisites

  • A Kestra instance with the DuckDB JDBC plugin (io.kestra.plugin.jdbc.duckdb).
  • A Parquet file available as a kestra:// internal storage URI or a namespace file URI.
  • A Parquet writer that records dictionary encodings in encodings, such as DuckDB. Writers that fall back in the middle of a column chunk can list a dictionary encoding next to PLAIN, and that fallback is not detected.

Secrets

  • None required: DuckDB runs in-process against files in task storage.

Quick start

  1. Import this blueprint into your Kestra instance.
  2. Execute it with parquet_uri set to a Parquet file in internal storage, and categorical_columns set to the columns that should be dictionary encoded.
  3. Read the verdict in the execution logs and download dictionary_encoding_report.csv from the task outputs.
  4. Raise max_degraded_columns if you accept some degraded columns, or keep it at 0 to fail on the first one.

How to extend

  • Trigger on landing: Add a Schedule or Webhook trigger with an inputs: block, or call this flow as a subflow after your ingestion flow.
  • Scan several files: Wrap the tasks in a ForEach over the files of a partition.
  • Alert on failure: Add a Slack or PagerDuty notification in a flow-level errors block.

Links

Orchestrate with Kestra
Orchestrate DuckDB with Kestra
Share this Blueprint
See How

New to Kestra?

Use blueprints to kickstart your first workflows.