Cache a computationally expensive task in an ETL pipeline using the taskCache property

Re-downloading a large, slow-changing dataset on every pipeline run wastes compute and hammers the source system. This blueprint shows how Kestra's taskCache property caches the output of an expensive extract task for a configurable TTL, so a 200k-row product catalog is fetched only once per day while fast-moving transaction data stays fresh. The cached catalog is then joined with new transactions in DuckDB to calculate profit per transaction, giving you a lean, repeatable ETL pattern that keeps downstream joins fast and the upstream database calm.

How it works

1. The transactions task (io.kestra.plugin.core.http.Download) downloads recent transaction data on every run, uncached, so the freshest sales records are always used.
2. The products task (io.kestra.plugin.core.http.Download) downloads the full product catalog and caches its output with taskCache set to enabled: true and ttl: PT24H. Within that 24-hour window Kestra reuses the cached result instead of re-downloading.
3. The duckdb task (io.kestra.plugin.jdbc.duckdb.Query) loads both files via inputFiles, runs a SQL JOIN on product_id with read_csv_auto, computes (sale_price - cost_price) * quantity AS profit, and stores the result with fetchType: STORE.

What you get

• A reusable caching pattern that skips redundant downloads of slow-changing reference data.
• Reduced load on the source system and lower run-to-run compute cost.
• In-pipeline analytics: a DuckDB join that enriches transactions and computes profit, with no external warehouse required.
• A stored output artifact you can pass to downstream tasks or flows.

Who it's for

• Data engineers building ETL pipelines that mix fast-changing and slow-changing sources.
• Analytics engineers who want lightweight, in-memory SQL transforms via DuckDB.
• Platform teams looking to cut cost and pressure on production databases.

Why orchestrate this with Kestra

DuckDB is an analytical engine, not a scheduler: it cannot decide when to run, when to skip redundant work, or how to recover from a failed extract. Kestra adds event and schedule triggers, declarative YAML you can version control, automatic retries, and full execution lineage across tasks. The taskCache property is the specific gap DuckDB cannot fill on its own: orchestration-level memoization that reuses an expensive task's output across runs based on a TTL, so you control freshness and cost from one place.

Prerequisites

• A running Kestra instance.
• The DuckDB plugin available (bundled in the default Kestra image).
• Outbound network access to the dataset URLs.

Secrets

This flow uses public dataset URLs and references no secret() values. To point it at a private source, add credentials as Kestra secrets and reference them with {{ secret('NAME') }}.

Quick start

1. Add this flow to a namespace in your Kestra instance.
2. Execute it once and confirm all three tasks succeed.
3. Run it again within 24 hours and check the execution: the products task is served from cache.
4. Open the duckdb task output to inspect the joined, profit-enriched result.

How to extend

• Tune ttl (for example PT1H or P1D) to match how often your reference data changes.
• Swap the Download tasks for database or object-storage extracts against your real sources.
• Add a Schedule trigger to run the pipeline on a cadence.
• Extend the DuckDB SQL with aggregations, window functions, or category-level rollups.
• Add downstream tasks to load the stored result into a warehouse or send alerts.

Links

• Plugin docs: https://kestra.io/plugins/plugin-jdbc-duckdb
• DuckDB orchestration: https://kestra.io/orchestration/duckdb