Query icon
Return icon
Switch icon
Log icon
SlackIncomingWebhook icon
Schedule icon

Arrow Flight SQL Endpoint TLS Compliance Gate

Probe an Arrow Flight SQL endpoint on a schedule and alert Slack when it answers without the TLS encryption your policy requires.

Categories
DataInfrastructure

Apache Arrow Flight SQL is a gRPC-based protocol that backends like Dremio and Apache DataFusion expose, usually as a shared analytics gateway serving several teams. Because it is shared infrastructure, nobody necessarily owns watching whether every client still connects over an encrypted channel. A connection string with encryption quietly flipped off can sit unnoticed for a long time. This blueprint probes the endpoint with a trivial query and separately confirms the way it was reached matches your own TLS policy, alerting Slack with the real host and port the moment it does not.

How it works

  1. probe_arrow_flight_endpoint (io.kestra.plugin.jdbc.arrowflight.Query) runs SELECT 1 AS probe, a trivial, backend-agnostic query, with allowFailure: true so a bad host, port, or credential does not stop the flow before it can be classified as PROBE_FAILED below.
  2. classify_endpoint_status (io.kestra.plugin.core.debug.Return) returns PROBE_FAILED when the probe did not return at all, TLS_POLICY_VIOLATION when it succeeded but use_encryption is false while require_tls is true, HEALTHY otherwise.
  3. route_by_endpoint_status (io.kestra.plugin.core.flow.Switch) branches on that status: HEALTHY logs it, the other two alert Slack with the real host, port, and encryption setting. A defaults branch logs anything unexpected.
  4. log_audit_summary always runs last. The errors block posts to Slack if the flow itself fails outside the gate logic above.

What you get

  • A real connectivity check against the actual endpoint.
  • A TLS compliance check against your own policy, not the driver's own default.
  • A Slack alert naming the real host and port, never a generic notice.
  • A fully local test path against an official reference Arrow Flight SQL server.

Who it's for

  • Platform and data infrastructure teams who run a shared Arrow Flight SQL gateway and need to know when a client reaches it without encryption.
  • Teams evaluating the Arrow Flight SQL JDBC plugin who want a working example of its connection string, independent of any specific backend's data model.

Why orchestrate this with Kestra

Arrow Flight SQL has no concept of alerting on how it was reached, that is a property of each client connection, not something the protocol surfaces on its own. Kestra adds the schedule, the policy comparison, and the Slack alert as one declarative, auditable flow.

Prerequisites

  • For local testing, the official reference Arrow Flight SQL server, which ships a small TPC-H dataset by default:

    docker run --name flight-sql -d -p 31337:31337 \
      -e TLS_ENABLED=1 \
      -e FLIGHT_PASSWORD=flight_password \
      voltrondata/flight-sql:latest
    

    With TLS_ENABLED=1 (this image's own default), the server serves a self-signed certificate, which is why disable_certificate_verification defaults to true here. Set it false against any real endpoint with a certificate from a trusted CA. The image hardcodes its username to flight_username; only the password is configurable via FLIGHT_PASSWORD. Note: this image is marked deprecated on Docker Hub in favor of voltrondata/sqlflite, but remains published and pullable as of this writing.

  • A Slack incoming webhook for alerts.

Secrets

  • ARROWFLIGHT_USERNAME, ARROWFLIGHT_PASSWORD: credentials for the endpoint. Against the reference server, the username is fixed to flight_username; the password is whatever FLIGHT_PASSWORD it was started with.

  • SLACK_WEBHOOK: Slack incoming webhook URL used by the alert tasks and the errors block.

    In Kestra OSS, set these as base64-encoded environment variables, for example SECRET_ARROWFLIGHT_USERNAME, SECRET_SLACK_WEBHOOK.

Inputs

Input Type Default Allowed values
arrow_flight_host STRING localhost any reachable Arrow Flight SQL host
arrow_flight_port INT 31337 any valid port
use_encryption BOOL true true, false
disable_certificate_verification BOOL true true, false
require_tls SELECT "true" "true", "false"

Outputs

  • {{ outputs.probe_arrow_flight_endpoint.row }}: the single probe row, when the query succeeded.
  • {{ outputs.classify_endpoint_status.value }}: HEALTHY, TLS_POLICY_VIOLATION, or PROBE_FAILED.

Quick start

  1. Start the reference server above and add the three secrets.
  2. Run this flow manually with the defaults (use_encryption: true, require_tls: "true"), it should report HEALTHY.
  3. Run it again with use_encryption: false, it should report TLS_POLICY_VIOLATION.
  4. Enable the every_30_minutes trigger.

How to extend

  • Point this gate at a Dremio coordinator's or a DataFusion server's own Arrow Flight SQL port to monitor a real shared gateway instead of the reference server.
  • Add a second probe with disable_certificate_verification: false against your own internal CA to also catch an expired or wrongly-issued certificate, not just encryption left off entirely.
  • Loop this flow with io.kestra.plugin.core.flow.Loop across a list of host:port endpoints to audit an entire fleet of Flight SQL gateways from one flow.

Pitfalls

  • disable_certificate_verification: true is meant for the local test server's self-signed certificate. Left true against a real endpoint, it silently defeats certificate validation, which this gate reports the same as a correctly validated connection whenever use_encryption is true.
  • TLS_POLICY_VIOLATION only fires when use_encryption is false. A connection that is encrypted but unauthenticated (disable_certificate_verification: true against an untrusted cert) is not currently distinguished from one that is both, see How to extend.
  • SELECT 1 AS probe is deliberately trivial so this gate works against any backend behind the protocol, confirming the endpoint answers and authenticates, not that any dataset behind it is healthy.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.