Data Ingestion Pipeline, Sync Hacker News Data to CSV Files Using CloudQuery

Build a CloudQuery data ingestion pipeline that extracts every Hacker News table and lands it on disk as structured CSV files, ready for analytics, exploration, or downstream processing. Instead of running CloudQuery by hand and wiring up your own scheduling, retries, and config management, this blueprint runs the CloudQuery CLI as a fully orchestrated Kestra task with inline source and destination configuration. It is a clean starting point for anyone evaluating CloudQuery as an extract engine or standardizing API to file ingestion as repeatable code.

How it works

1. The hn_to_parquet task uses io.kestra.plugin.cloudquery.CloudQueryCLI to run the CloudQuery CLI directly inside Kestra.
2. An inline inputFiles entry, config.yml, defines both the CloudQuery source and destination as a single multi-document YAML spec.
3. The source spec points to cloudquery/hackernews (v3.7.7), syncing all tables (tables: ["*"]) with item_concurrency: 100 and an incremental start_time derived from {{ execution.startDate | dateAdd(-1, 'DAYS') }}.
4. The destination spec uses cloudquery/file (v5.4.6) with format: csv, writing one path per table using the CloudQuery {{TABLE}} and {{UUID}} placeholders. These are wrapped in raw blocks so Kestra does not parse them as Pebble expressions.
5. The commands block runs cloudquery sync config.yml, and outputFiles captures every generated **/*.csv file as a Kestra output for use by downstream tasks.

What you get

• A complete, runnable CloudQuery sync defined entirely in declarative YAML.
• All Hacker News tables exported as CSV, partitioned per table.
• Incremental ingestion that pulls only the last day of data on each run.
• CSV outputs surfaced as Kestra task outputs for chaining into other flows.

Who it's for

• Data engineers standardizing API to file or API to warehouse ingestion.
• Analytics engineers who want raw Hacker News data without writing custom extractors.
• Teams evaluating CloudQuery and looking for an orchestrated reference pattern.

Why orchestrate this with Kestra

The CloudQuery CLI syncs data, but it has no built in scheduler, retry policy, dependency graph, or run history. Kestra adds event and schedule triggers, automatic retries, execution logs, output lineage, and a declarative YAML definition that lives in version control. You get observability and reusability across many syncs instead of a one off command on a cron line that nobody can audit.

Prerequisites

• A Kestra instance with the CloudQuery plugin available.
• A CloudQuery API key (set in the task env as CLOUDQUERY_API_KEY).

Secrets

• CLOUDQUERY_API_KEY: the CloudQuery API key used to authenticate connectors. The task env reads it as {{ secret('CLOUDQUERY_API_KEY') }}, so store the key in your secret backend rather than hardcoding it.

Quick start

1. Add the CLOUDQUERY_API_KEY secret to your Kestra instance with your own key (see the CloudQuery introduction blog post).
2. Adjust the namespace to match your environment.
3. Save and execute the flow.
4. Inspect the CSV files attached as outputs of the hn_to_parquet task.

How to extend

• Swap cloudquery/hackernews for any other CloudQuery source plugin to ingest a different system.
• Change the destination from cloudquery/file to a warehouse such as BigQuery, Snowflake, or PostgreSQL, or to cloud storage.
• Switch format: csv to Parquet or JSON for analytics ready outputs.
• Add a Schedule trigger and downstream tasks to load the CSV into a database or run dbt transformations.

Links

• Kestra CloudQuery plugin documentation
• Introduction to CloudQuery (Kestra blog)