Arrow Flight SQL Endpoint TLS Compliance Gate
Probe an Apache Arrow Flight SQL endpoint and alert Slack when it answers over an unencrypted connection in violation of your own TLS policy.
Audit DataHub metadata ingestion health on a schedule, classify runs by exitCode and stderr, and alert Slack when the catalog goes stale.
A data catalog is only as trustworthy as its last successful ingestion. When a DataHub
ingestion recipe silently starts failing, swallowing a bad connector password, a schema
change, or a network change, nobody notices until an analyst finds a dataset with no owner,
no lineage, and no fresh schema. This blueprint turns a DataHub ingestion run into a Day 2
governance gate: it runs the recipe, reads the exact exitCode and stdErrLineCount the
DataHub ingestion CLI reports, classifies the run, and alerts Slack before stale metadata
becomes everyone else's problem.
route_ingestion_mode (io.kestra.plugin.core.flow.If) checks the dry_run input.
When true it runs probe_console, when false it runs probe_live. Both are
io.kestra.plugin.datahub.Ingestion tasks with allowFailure: true so a failed probe
does not stop the flow before it can be classified and alerted on.probe_console uses recipe sink type: console, metadata is printed to stdout only,
nothing is written to the catalog.probe_live uses recipe sink type: datahub-rest with config.server pointed at
datahub_gms_server, metadata is written to the live catalog.
Both use io.kestra.plugin.scripts.runner.docker.Docker as the taskRunner with
networkMode set to docker_network, so the ingestion container can resolve the
source database and the GMS server by container name.classify_ingestion_status (io.kestra.plugin.core.debug.Return) reads exitCode and
stdErrLineCount from whichever probe ran (guarded with is defined, since a probe
that never started has no output at all) and returns one status string: PROBE_FAILED
(no output at all), FAILED (non-zero exitCode), DEGRADED (zero exitCode but
stdErrLineCount over stderr_warning_threshold), or HEALTHY.route_by_ingestion_status (io.kestra.plugin.core.flow.Switch) branches on that status:log_healthy (io.kestra.plugin.core.log.Log) logs it, nothing else happens.io.kestra.plugin.slack.notifications.SlackIncomingWebhook task posts a message built
only from the probe's own exitCode or stdErrLineCount, never an invented number.log_audit_summary (io.kestra.plugin.core.log.Log) always runs last and logs the final
status, the source type, dry_run, and the GMS server, closing out the audit cycle.errors block posts to Slack if the flow itself fails outside the gate logic above.HEALTHY, DEGRADED, FAILED, PROBE_FAILED) built only
from fields the DataHub ingestion CLI actually returns: exitCode and stdErrLineCount.console sink) so you can validate connectivity and
recipe correctness before ever writing to the catalog.io.kestra.plugin.datahub.Ingestion with both a console and a datahub-rest sink.The DataHub ingestion CLI runs a recipe and exits, it has no concept of a schedule, a threshold, or an alert. Kestra adds the schedule, the dry run and live modes behind one input, the classification logic that turns a raw exit code into an operational signal, the conditional Slack alerting, and a full execution history of every ingestion run next to every other pipeline you orchestrate. The whole gate is one declarative YAML file you can version and review like code.
A local DataHub instance. The standard way to run one locally is the DataHub CLI
quickstart, which manages a Docker Compose stack of DataHub's own containers
(pip install acryl-datahub first):
DATAHUB_MAPPED_GMS_PORT=8081 datahub docker quickstart
This maps datahub-gms to host port 8081 instead of its default 8080, which otherwise
collides with Kestra's own default UI port. The containers join a Docker network named
datahub_network.
Connect your Kestra container to that network so the ingestion task's Docker task
runner can resolve datahub-gms and your source database by name. Run
docker network ls first to confirm the exact network name on your machine, then:
docker network connect datahub_network YOUR_KESTRA_NETWORK
A source database reachable on the same network, for example:
docker run -d --name postgres --network datahub_network -e POSTGRES_PASSWORD=<password> -e POSTGRES_DB=datahub_source postgres:16
A Slack incoming webhook for alerts.
DATAHUB_SOURCE_PASSWORD: password for the source database the ingestion recipe reads
from. In Kestra OSS, set it as the base64-encoded environment variable
SECRET_DATAHUB_SOURCE_PASSWORD.SLACK_WEBHOOK: Slack incoming webhook URL used by the alert tasks and the errors
block. In Kestra OSS, set it as the base64-encoded environment variable
SECRET_SLACK_WEBHOOK.| Input | Type | Default | Allowed values |
|---|---|---|---|
source_type |
SELECT | postgres |
postgres, mysql |
source_host_port |
STRING | postgres:5432 |
any host:port |
source_database |
STRING | datahub_source |
any database name |
source_username |
STRING | datahub |
any username |
datahub_gms_server |
STRING | http://datahub-gms:8080 |
any GMS URL |
docker_network |
STRING | datahub_network |
any Docker network name |
container_image |
STRING | acryldata/datahub-ingestion:head |
any DataHub ingestion image |
stderr_warning_threshold |
INT | 5 |
any non-negative integer |
dry_run |
BOOL | true |
true, false |
DATAHUB_MAPPED_GMS_PORT=8081 datahub docker quickstart and
a source database container on the same datahub_network.datahub_network.DATAHUB_SOURCE_PASSWORD and SLACK_WEBHOOK secrets.dry_run: true and read log_audit_summary
for the classified status.dry_run to false once the dry run reports HEALTHY, then enable the
six_hourly trigger.outputs.probe_live.vars or outputs.probe_console.vars for any structured counts
your recipe's source.config is set up to emit, and fold them into the classification.source_type value for another DataHub source and branch the recipe config
block per type if the fields differ.stderr_warning_threshold check with a comparison against the previous
run's stdErrLineCount stored in a Kestra KV entry, to catch a slow drift instead of a
fixed threshold.HEALTHY that queries the DataHub GraphQL API to assert a
specific dataset's lastIngested timestamp moved forward.datahub-gms container defaults to host port 8080, the same default port
Kestra's own UI uses. Remap it with DATAHUB_MAPPED_GMS_PORT before starting Kestra on
the same host.PROBE_FAILED instead of a classified exit code.console sink accepts no config block at all, adding one is a silent no-op at
best and a recipe validation error at worst. Only datahub-rest needs config.server.