Faktory Job Enqueue Path Reachability Sentinel

A Faktory server exposes two independent subsystems: the raw Faktory Work Protocol (FWP) TCP port that real workers and job producers depend on, and a separate Web UI HTTP port used for dashboards and monitoring. Checking only one - as this repository's existing Faktory blueprint does, which polls just the Web UI /stats endpoint - can miss a failure in the other. This blueprint pushes a synthetic canary job over the real FWP enqueue path and cross-checks it against the Web UI's own reporting, catching the moment either subsystem breaks or the two fall out of sync with each other.

How it works

1. periodic_enqueue_check (io.kestra.plugin.core.trigger.Schedule) runs every 15 minutes, explicitly passing faktory_host, faktory_port, faktory_web_stats_url, and canary_queue through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
2. probe_push_canary_job (io.kestra.plugin.faktory.Push) pushes a synthetic job over the raw FWP TCP socket to a dedicated, worker-free canary queue, capturing the returned job id (jid).
3. probe_fetch_stats (io.kestra.plugin.core.http.Request) reads the Web UI's /stats endpoint - a completely separate HTTP subsystem - and captures the raw JSON response body.
4. evaluate_enqueue_reachability (io.kestra.plugin.core.flow.If) branches on whether the canary queue's reported depth (parsed from the stats response with jq) is greater than zero.
   • then (HEALTHY): log_enqueue_healthy records the pass with the canary job id.
   • else (ENQUEUE_PATH_UNHEALTHY): alert_enqueue_unhealthy reports the mismatch to Slack.
5. log_audit_complete always runs last, printing the canary job id and queue regardless of which branch fired.
6. The errors block alerts Slack separately if either probe fails outright - an unreachable FWP port or an unreachable Web UI.

What you get

• A cross-subsystem consistency check that this repository's existing Faktory blueprint cannot provide: it only ever polls the Web UI, so a healthy dashboard with a silently broken FWP enqueue port would read as HEALTHY there, but is caught here.
• A worked example of io.kestra.plugin.faktory.Push, a real task class with zero prior usage anywhere in this repository's blueprints, combined with the plugin's documented /stats JSON schema (faktory.queues, verified directly from Faktory's own Go source) parsed via jq on a raw HTTP response body - the officially documented pattern for this exact situation, per Kestra's own HTTP request documentation.
• A non-destructive design: the canary queue has no real consumer by design, so the pushed job never gets executed, retried, or sent to the dead queue (retries: -1 discards it immediately in the unlikely case something does pick it up).

Who it's for

• Platform and backend teams running Faktory who want proof that the actual job-submission path works, not just that the dashboard responds.
• Teams who have been burned by a worker-side outage that a Web-UI-only health check missed because the dashboard itself stayed up.
• Anyone evaluating the Faktory plugin who wants a worked example of Push used for synthetic monitoring rather than real job submission.

Why orchestrate this with Kestra

Faktory's own Web UI has no built-in cross-check between its FWP and HTTP subsystems, and no Slack integration of its own. Kestra supplies the schedule, the HEALTHY/ENQUEUE_PATH_UNHEALTHY classification as a first-class branch, and an execution history showing exactly when the two subsystems first disagreed.

Prerequisites

• A running Faktory server, reachable over both its FWP port (default 7419) and its Web UI port (default 7420). Faktory publishes an official open-source Docker image requiring no account or license - docker run --rm -p 7419:7419 -p 7420:7420 contribsys/faktory:latest starts a local instance for testing.
• A dedicated queue name with no real worker subscribed to it (canary_queue).
• A Slack incoming webhook for alerts.

Secrets

• FAKTORY_PASSWORD: password for io.kestra.plugin.faktory.Push, matching the plugin's own documented password connection property. Leave the secret empty for an unauthenticated local Faktory instance.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_enqueue_unhealthy and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_FAKTORY_PASSWORD=$(echo -n '<your-password>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

• faktory_host (STRING, default localhost): Faktory server host for the FWP connection.
• faktory_port (INT, default 7419): FWP TCP port.
• faktory_web_stats_url (STRING, default http://localhost:7420/stats): Web UI stats endpoint, a separate port and protocol.
• canary_queue (STRING, default kestra-canary-queue): the dedicated, worker-free queue the canary job is pushed to.

Outputs

• outputs.probe_push_canary_job.jid (String): the Faktory job id of the pushed canary (verified field from Push's own Output class).
• outputs.probe_fetch_stats.body (String): the raw /stats JSON response, parsed with jq for the faktory.queues map.

Quick start

1. Run a local Faktory server (docker run --rm -p 7419:7419 -p 7420:7420 contribsys/faktory:latest) or point this at a real one.
2. Add the SLACK_WEBHOOK_URL secret (and FAKTORY_PASSWORD if the server requires authentication).
3. Run the flow manually and confirm log_enqueue_healthy fires, with the pushed jid visible in the Faktory Web UI's queue list for canary_queue.
4. Temporarily point faktory_web_stats_url at an unreachable address and re-run to confirm the errors block fires, distinct from the ENQUEUE_PATH_UNHEALTHY branch.
5. Enable periodic_enqueue_check once you trust the check against your real Faktory server.

How to extend

• Loop over several canary_queue values (one per priority tier your real queues use) to verify the enqueue path separately for each, catching a tier-specific FWP routing issue.
• Add a total_enqueued trend check using the same /stats response (also verified from Faktory's own DataSnapshot struct) to catch a server-wide enqueue slowdown, not just a per-queue one.
• Add a remediation branch that pages a different on-call rotation when the errors block fires (infrastructure-down) versus when ENQUEUE_PATH_UNHEALTHY fires (subsystem desync) - the two failure modes likely need different responders.

Pitfalls

• canary_queue must never be a queue a real worker also consumes - if a worker picks up the synthetic KestraCanaryHeartbeat job type and doesn't recognize it, retries: -1 ensures it is discarded immediately rather than retried indefinitely, but this sentinel is designed around the queue having no consumer at all.
• The Web UI /stats body is a raw string, not pre-parsed JSON - verified directly from Kestra's own HTTP request documentation ("the expression returns a string, not a JSON"), which is why jq is used here rather than direct dot-notation field access.
• A canary job pushed moments before this sentinel's next run could still be counted if nothing ever drains canary_queue - queue depth only grows across repeated runs unless something clears it periodically; this is expected and does not indicate a problem, but a very long-running deployment may want an occasional manual purge of the canary queue.
• This blueprint is UNTESTED against a live Faktory server. The Push task's properties, the /stats JSON schema (faktory.queues, total_enqueued, etc.), and the FWP/Web-UI port distinction are all taken verbatim from the Kestra plugin's own source and Faktory's own Go server source - but no Kestra engine was run to execute this flow end to end.

Links

• Kestra Faktory plugin documentation
• Faktory official Docker image
• Kestra HTTP request documentation (body is a string, parse with jq)
• Kestra secrets (OSS vs Enterprise)