Build an ETL Pipeline from Cassandra to BigQuery Using CSV Export

Move operational data out of Apache Cassandra and into Google BigQuery for analytics, reporting, and downstream data warehouse workloads. This blueprint runs a Cassandra to BigQuery ETL pipeline that queries a Cassandra table with CQL, serializes the results to CSV, and batch loads the file into a BigQuery table. It closes the gap between a distributed NoSQL operational store and a columnar analytical warehouse so you can run SQL analytics on data that lives in Cassandra.

How it works

1. The query_cassandra task (io.kestra.plugin.cassandra.Query) connects to a Cassandra cluster endpoint and runs a cql SELECT against the source table, with fetchType: STORE to persist the full result set to Kestra internal storage.
2. The write_to_csv task (io.kestra.plugin.serdes.csv.IonToCsv) reads the stored Ion result from outputs.query_cassandra.uri and converts it into a CSV file.
3. The load_bigquery task (io.kestra.plugin.gcp.bigquery.Load) loads the CSV from outputs.write_to_csv.uri into destinationTable using format: CSV, with csvOptions setting fieldDelimiter and skipLeadingRows: 1 to skip the header row.

What you get

• A reproducible, end to end NoSQL to warehouse ETL pipeline in declarative YAML.
• Clean separation of extract (CQL), transform (serialization to CSV), and load (BigQuery) steps.
• Result sets passed between tasks through Kestra internal storage instead of local disk.
• A pattern you can point at any Cassandra table and BigQuery dataset.

Who it's for

• Data engineers consolidating operational NoSQL data into a central warehouse.
• Analytics teams that need Cassandra data queryable in BigQuery.
• Platform teams standardizing batch ETL across heterogeneous data stores.

Why orchestrate this with Kestra

Cassandra and BigQuery each ship clients and load utilities, but neither owns the cross system workflow. Kestra orchestrates the full extract, transform, and load sequence: event or schedule triggers to launch loads, automatic retries on transient cluster or API failures, execution lineage across all three tasks, and passing query output between steps without glue scripts. The whole pipeline stays declarative YAML, versioned and reviewable, which the CQL shell and the bq loader cannot coordinate on their own.

Prerequisites

• A reachable Apache Cassandra cluster and a source table.
• A Google Cloud project with a BigQuery dataset and destination table (or extend the load task with a schema to create it).
• A GCP service account with BigQuery load permissions.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: the GCP service account JSON key used by the load_bigquery task to authenticate to BigQuery.

Quick start

1. Add the GCP_SERVICE_ACCOUNT_JSON secret to your Kestra instance.
2. Point the Cassandra session endpoints and localDatacenter at your cluster and adjust the cql query.
3. Set projectId and destinationTable on the load_bigquery task to your BigQuery dataset and table.
4. Run the flow and confirm rows land in BigQuery.

How to extend

• Add a schedule or event trigger to run the load on a cadence or on arrival of new data.
• Provide a schema on the load task to auto create the destination table.
• Swap CSV for Avro or Parquet serialization for larger or typed datasets.
• Add a post load transformation or dbt task to model the warehouse tables.

Links

• Google Cloud Platform plugin docs
• Cassandra plugin docs
• BigQuery orchestration with Kestra