Run a Papermill notebook

Run a Jupyter notebook as a first-class, reproducible step in a data pipeline using Papermill inside a Kestra Python task. Notebooks are great for analysis and prototyping, but they are hard to operationalize: they sit on a laptop, run interactively, and leave no record of what executed. This blueprint solves that by executing a notebook headlessly with Papermill, capturing the fully rendered output notebook as a Kestra output artifact, and making the whole run versioned, observable, and ready to schedule.

How it works

The flow has a single io.kestra.plugin.scripts.python.Commands task named python.

1. namespaceFiles is enabled, so the notebook stored in your namespace under src/example.ipynb.py is synced into the task working directory.
2. beforeCommands install the runtime dependencies: pip install ipykernel papermill, then register a Jupyter kernel with python -m ipykernel install --name python3.12.0.
3. The main command papermill src/example.ipynb.py output.ipynb -k python3.12.0 executes the input notebook cell by cell against that kernel and writes the rendered result to output.ipynb.
4. outputFiles captures output.ipynb, so the executed notebook (with all cell outputs and results inlined) is stored as a downloadable Kestra output.

What you get

• Headless, repeatable notebook execution with no manual "Run all".
• The fully executed output.ipynb saved as a Kestra output artifact for audit and review.
• A pinned kernel (python3.12.0) so runs are reproducible.
• Dependencies installed per run via beforeCommands, keeping the environment explicit.

Who it's for

• Data scientists who want their notebooks to run on a schedule instead of by hand.
• Data and analytics engineers operationalizing exploratory notebooks into pipelines.
• ML practitioners running parameterized reporting or batch notebooks.

Why orchestrate this with Kestra

Papermill executes a notebook but does not schedule it, retry it, or track its history. Kestra adds event and schedule triggers, automatic retries on failure, execution-level outputs and lineage, and a declarative YAML definition you can version in Git. The executed notebook is captured as an output rather than left on disk, and Namespace Files keep the notebook source synced without manual copying. That is the operational layer Papermill alone cannot provide.

Prerequisites

• A Kestra instance with the Python script plugin available.
• A Docker or local task runner that can pip install packages.
• A Jupyter notebook uploaded as a Namespace File at src/example.ipynb.py.

Secrets

This flow references no secrets. Add them with {{ secret('NAME') }} only if your notebook needs credentials to reach external systems.

Quick start

1. Upload your notebook to the namespace as src/example.ipynb.py.
2. Copy this flow into your namespace.
3. Execute the flow and watch the python task install dependencies and run Papermill.
4. Download output.ipynb from the execution outputs to review the rendered notebook.

How to extend

• Add inputs and pass them to Papermill with -p name value to parameterize each run.
• Add a io.kestra.plugin.core.trigger.Schedule trigger to run the notebook on a cron schedule.
• Convert the result with jupyter nbconvert to publish an HTML or PDF report.
• Push output.ipynb to object storage or send a notification when the run completes.

Links

• Plugin docs: https://kestra.io/plugins/plugin-script-python