Ingest Postgres Metadata into DataHub on a Schedule

A data catalog is only useful while it matches reality, and it only matches reality if something re-syncs it on a schedule. This blueprint runs the standard DataHub ingestion recipe through io.kestra.plugin.datahub.Ingestion: a postgres source scans tables, views, and schemas from the database you point it at, and a datahub-rest sink pushes the metadata to your DataHub GMS server. Slack hears about every sync and, separately, about every failure, so the catalog never goes quietly stale.

How it works

1. ingest_postgres_metadata (io.kestra.plugin.datahub.Ingestion) executes the DataHub ingestion CLI inside its container image against the inline recipe. The recipe is the exact YAML DataHub documents, a source block and a sink block, so any recipe you already run with datahub ingest drops in unchanged.
2. The postgres source connects with credentials pulled from Kestra secrets and extracts table, view, and schema metadata. Profiling is disabled here to keep the sync fast; enable it when you want column statistics in the catalog.
3. The datahub-rest sink writes to the GMS server, with both the server URL and the access token resolved from secrets at runtime, never stored in the flow.
4. notify posts a confirmation to Slack naming the database and execution, and the errors block posts a distinct alert when the ingestion fails.
5. A disabled-by-default Schedule trigger re-runs the sync every morning at 05:00.

What you get

• A catalog that refreshes itself daily instead of drifting from the real schema.
• Credentials, GMS URL, and token all resolved from secrets, so the flow is safe to commit.
• The unmodified DataHub recipe format, meaning every source option in the DataHub docs applies as-is.
• Host and database as flow inputs, so one flow definition covers every environment.

Who it's for

• Data platform teams running DataHub who currently trigger datahub ingest from cron or by hand.
• Governance owners who need proof, in execution history, of when each sync ran and whether it succeeded.
• Analytics engineers tired of answering whether the catalog reflects last week's schema changes.

Why orchestrate this with Kestra

The DataHub CLI runs a recipe once; keeping a catalog trustworthy means running it every day, retrying transient GMS or database failures, and telling someone when it breaks. Kestra wraps the unchanged recipe with scheduling, secret management, per-run history, and alerting. When the sync fails, the Slack alert and the execution logs point straight at the failing source or sink instead of a cron job failing silently.

Prerequisites

• A running DataHub instance with its GMS endpoint reachable from Kestra.
• A Postgres user with read access to the schemas you want cataloged.
• Docker available to the Kestra worker, since the task runs the acryldata/datahub-ingestion container image.
• A Slack incoming webhook for sync reports.

Secrets

• DATAHUB_GMS_URL: URL of the DataHub GMS server, for example http://datahub-gms:8080.
• DATAHUB_TOKEN: DataHub personal access token with permission to ingest metadata.
• POSTGRES_USERNAME: Postgres user for metadata extraction.
• POSTGRES_PASSWORD: password for that user.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the five secrets to your Kestra namespace.
2. Set the postgres_host and postgres_database inputs to a real database and execute the flow.
3. Open DataHub and confirm the datasets appear, then set disabled: false on the daily trigger.

How to extend

• Scope the sync with schema_pattern or table_pattern in the source config, as shown in the post-migration ingestion blueprint.
• Enable profiling to add row counts and column statistics to the catalog.
• Swap the source type to mysql, snowflake, or any other DataHub source; the flow shape stays identical.
• Fan the pattern out across several databases with ForEach, as shown in the multi-source catalog sync blueprint.

Links

• DataHub plugin documentation
• DataHub metadata ingestion documentation