Run a ZenML flow from Kestra until its completion and stream logs in real-time

Trigger a ZenML machine learning pipeline directly from Kestra and let Kestra own the scheduling, execution, and observability around it. ZenML structures your ML code into reusable steps and pipelines, but it leaves orchestration, scheduling, retries, and cross-system dependencies to you. This blueprint closes that gap: it runs your ZenML pipeline inside a container on a daily schedule, streams the logs back into Kestra, and gives you a single place to operate model training alongside the rest of your data platform.

How it works

The flow has one task, run_zenml, of type io.kestra.plugin.scripts.python.Commands. It launches the zenmldocker/zenml-server container image and runs python zenml_flow.py. The pipeline script is supplied through inputFiles and defines a ZenML @pipeline with two @step functions: load_data returns a value and train_model consumes it. The task connects to your ZenML server through three environment variables: ZENML_STORE_API_KEY (sourced from a secret), ZENML_STORE_URL, and ZENML_STACK. A io.kestra.plugin.core.trigger.Schedule trigger fires the flow every day at 9 AM using the cron expression 0 9 * * *.

What you get

• A ZenML pipeline that runs unattended on a daily schedule.
• Real-time log streaming from the container into the Kestra UI.
• A reproducible container image so the ML environment is identical on every run.
• A clear separation between ML logic (the ZenML script) and orchestration (the Kestra flow).

Who it's for

• ML engineers who write ZenML pipelines and need them to run reliably on a schedule.
• Data and platform teams who want ML training to live in the same orchestrator as their ELT and analytics jobs.
• Teams moving ZenML from local experiments to scheduled, observable production runs.

Why orchestrate this with Kestra

ZenML defines pipelines and steps, but it does not provide a full-featured scheduler, event triggers, or cross-tool dependency management on its own. With Kestra you get scheduled and event-driven triggers, automatic retries, execution history and lineage, and declarative YAML that keeps the pipeline definition in version control. You can chain the ZenML run to upstream data loads and downstream deployment or notification steps, all in one place, instead of stitching together ZenML's runtime with an external cron.

Prerequisites

• A running ZenML server reachable from the task (for example via Docker Compose with a MySQL backend).
• A ZenML stack configured (the flow uses default).
• Docker available to Kestra so it can pull and run zenmldocker/zenml-server.

Secrets

• ZENML_API_KEY: the ZenML service account API key, referenced as {{ secret('ZENML_API_KEY') }} and passed to the container as ZENML_STORE_API_KEY.

Quick start

1. Start a ZenML instance, for example with a Docker Compose stack pairing zenmldocker/zenml-server with MySQL.
2. Open the ZenML dashboard at http://localhost:4200, create a user, and generate an API key under Settings, Service Accounts.
3. Store that key in Kestra as the ZENML_API_KEY secret.
4. Adjust ZENML_STORE_URL and ZENML_STACK to match your environment.
5. Add the blueprint and let the daily schedule run it, or trigger an execution manually to verify.

How to extend

• Replace the inline zenml_flow.py with your own pipeline, or move it to a namespace file with namespaceFiles.enabled: true and edit it in the Files editor.
• Change the cron expression on the Schedule trigger, or swap in an event-based trigger.
• Attach a taskRunner (such as io.kestra.plugin.ee.aws.runner.Batch on AWS Fargate) to run the training on cloud compute instead of locally.
• Add downstream tasks to register the model, run evaluation, or send a notification when training completes.

Links

• plugin-script documentation