Validate Data Quality with Pandera in Python

This flow runs a Pandera schema validation inside a Python Script task. Define a typed, rule-based schema once, Pandera checks every column, constraint, and data-type expectation on every run and halts the pipeline before bad data reaches any downstream system.

How it works

1. A daily Schedule trigger starts the flow at 09:00 every morning.
2. validate_data (io.kestra.plugin.scripts.python.Script, python:3.11-slim) installs pandas and pandera via dependencies, then:
   • Builds (or loads) a DataFrame, replace the inline sample with any real source: a database query, an S3 download, a Kestra input file, etc.
   • Declares a pa.DataFrameSchema that names every expected column, its Python type, and any value-level checks (e.g. price > 0, inventory >= 0).
   • Calls schema.validate(data, lazy=True) so all failing columns are reported at once rather than stopping at the first error.
   • On failure, prints the failure_cases DataFrame and exits with code 1, which marks the task, and the execution, as FAILED.
3. parallel_post_processing (io.kestra.plugin.core.flow.Parallel) only runs when validation passes, firing success_notification and trigger_downstream_etl simultaneously. Replace the Log tasks with real notifications (Slack, email) or downstream flow triggers.

What you get

• A single, version-controlled schema that acts as a living data contract.
• Lazy validation: every failing column is reported in one shot instead of failing on the first error.
• A hard pipeline stop on bad data, nothing is loaded until the data is clean.
• Easy extension: swap the inline DataFrame for any real data source without touching the schema definition.

Who it's for

Data engineers and analytics engineers who want a lightweight, code-native quality gate before a dbt run, a warehouse load, or any batch pipeline, without standing up a separate quality platform.

Why orchestrate this with Kestra

• The dependencies field keeps the container image minimal: Pandera is installed on demand, pinned per flow, and isolated from other tasks.
• Scheduling, retries, alerting, and audit logs are handled by Kestra, the Python task stays pure business logic.
• Outputs from the validation script can be forwarded to downstream tasks or used in conditional branches with io.kestra.plugin.core.flow.If.

Prerequisites

• A Kestra worker able to run Docker containers (python:3.11-slim image).
• No secrets are required for the demo run.

Secrets

Add secrets when replacing the inline data with a real source, for example:

• POSTGRES_URL, POSTGRES_USER, POSTGRES_PASSWORD, to query a database.
• AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, to download a file from S3.

Quick start

1. Import the flow and click Execute. The demo DataFrame passes all constraints and you will see Data Quality Check Passed! in the Logs.
2. Edit the script: change one price value to -0.5 and run again. Pandera reports the failure case and the execution fails, so nothing downstream runs.
3. Replace the data = pd.DataFrame(…) block with your real data source and update the schema columns and checks to match.
4. Adjust the cron expression on the daily trigger to fit your schedule.

How to extend

• Add an io.kestra.plugin.core.flow.If after validate_data to branch on outputs.validate_data.exitCode and send a Slack or email alert on failure.
• Store the schema in a namespace file and load it with inputFiles so non-engineers can update column rules without touching the flow.
• Chain multiple validate_data tasks (one per table) inside a io.kestra.plugin.core.flow.Parallel block to validate an entire dataset family in one execution.
• Pin the Pandera version inside dependencies (e.g. pandera==0.20.4) for reproducible results across environments.

Links

• Pandera documentation: https://pandera.readthedocs.io
• Pandera checks reference: https://pandera.readthedocs.io/en/stable/checks.html
• Python scripts plugin: https://kestra.io/plugins/plugin-script-python
• Kestra namespace files: https://kestra.io/docs/concepts/namespace-files