Load a scheduled Apify web scrape into Postgres
Run an Apify actor on a schedule, fetch its dataset, reshape the records, and bulk-load them into Postgres with cost caps and Slack alerts.
Check Parquet column chunk metadata with DuckDB for categorical columns that fell back from dictionary to PLAIN encoding, and fail when too many degrade.
Checks a Parquet file for silent loss of dictionary encoding before it is loaded into a lakehouse or warehouse. Categorical columns such as status codes, country codes and payment methods normally compress through dictionary encoding. When a producer writes high-cardinality values into one of them (random tokens, UUIDs, unparsed timestamps), the writer's dictionary page overflows and the column falls back to PLAIN encoding. File size grows, filter pushdown slows down, and no row count or file size check shows it.
The flow reads parquet_metadata() with embedded DuckDB, groups the column chunks by column across all row groups, and reports every column you listed as dictionary encoded in all row groups, partly or not at all. It writes a per-column CSV and fails the execution when the number of degraded columns is above your limit, or when a listed column does not exist in the file. It only reads metadata and never modifies the file.
profile_dictionary_encoding (io.kestra.plugin.jdbc.duckdb.Query, fetchType: FETCH_ONE) mounts parquet_uri with inputFiles and aggregates parquet_metadata() per column. A column counts as healthy when every row group has a dictionary encoding in its encodings, and as degraded when at least one row group does not. Columns you listed that are missing from the file are counted separately, so a typo cannot pass silently.HEALTHY, DEGRADED_WITHIN_TOLERANCE, CRITICAL_DICTIONARY_OVERFLOW or COLUMNS_NOT_FOUND.write_encoding_report (io.kestra.plugin.jdbc.duckdb.Queries) writes dictionary_encoding_report.csv with, for each listed column, the row group count, the number of dictionary-encoded row groups, uncompressed and compressed bytes, the compression ratio and a status.route_by_encoding_health (io.kestra.plugin.core.flow.Switch) acts on the verdict:HEALTHY: logs the result.DEGRADED_WITHIN_TOLERANCE: logs a warning that names the degraded columns.CRITICAL_DICTIONARY_OVERFLOW: fail_on_dictionary_overflow (Fail) fails the execution.COLUMNS_NOT_FOUND: fail_on_missing_columns (Fail) fails the execution.The check is cheap but easy to forget. In Kestra it runs right after a file lands, leaves a per-column report as an execution output and fails the execution so downstream flows can stop. A Switch keeps the policy in one visible place.
io.kestra.plugin.jdbc.duckdb).kestra:// internal storage URI or a namespace file URI.encodings, such as DuckDB. Writers that fall back in the middle of a column chunk can list a dictionary encoding next to PLAIN, and that fallback is not detected.parquet_uri set to a Parquet file in internal storage, and categorical_columns set to the columns that should be dictionary encoded.dictionary_encoding_report.csv from the task outputs.max_degraded_columns if you accept some degraded columns, or keep it at 0 to fail on the first one.Schedule or Webhook trigger with an inputs: block, or call this flow as a subflow after your ingestion flow.ForEach over the files of a partition.errors block.