Use Redis List Realtime Trigger to push events into Cassandra

Stream events from a Redis List into Apache Cassandra the instant they arrive, with no polling and no batch lag. This blueprint wires a Redis List realtime trigger to a Cassandra insert so each JSON product record pushed onto a Redis key is captured and written to a Cassandra table immediately. It solves the classic ingestion gap where a fast in-memory queue produces events but you need a durable, queryable store to land them in, without standing up a bespoke consumer service.

How it works

1. The io.kestra.plugin.redis.list.RealtimeTrigger watches the Redis List at key products on the redis://localhost:6379/0 connection and fires the flow the moment a new element is pushed (for example via LPUSH).
2. The insert_into_cassandra task (io.kestra.plugin.cassandra.Query) connects to the Cassandra endpoint on localhost:9042 in datacenter datacenter1 and runs an INSERT into kestra.products.
3. The trigger payload is parsed inline with jq expressions such as {{ trigger.value | jq(".product_id") | first }} to map product_id, product_name, product_category, and brand into table columns.

What you get

• Real-time, event-by-event ingestion from Redis into Cassandra
• A declarative pipeline with zero custom consumer code
• Inline JSON-to-column mapping using jq
• A reproducible local setup using Docker for both Redis and Cassandra

Who it's for

• Data engineers building streaming ingestion pipelines
• Platform teams bridging in-memory queues to durable stores
• Developers prototyping event-driven architectures on Redis and Cassandra

Why orchestrate this with Kestra

Redis itself has no scheduler or workflow engine: it can hold a list but cannot react to pushes, retry a failed downstream write, or give you execution history. Kestra adds an event trigger that fires on each Redis push, automatic retries on transient Cassandra errors, full execution lineage and logs per event, and a declarative YAML definition you can version and review. You get the responsiveness of a streaming consumer with the observability and governance of an orchestrated flow.

Prerequisites

• A running Redis instance reachable at redis://localhost:6379/0
• A running Cassandra instance reachable at localhost:9042 with datacenter datacenter1
• A kestra keyspace and products table created in Cassandra

Secrets

This flow connects to local Redis and Cassandra over plain connection strings and does not reference any {{ secret(...) }} values. For production, move credentials and endpoints into Kestra secrets and reference them in the trigger url and the task session.

Quick start

1. Start Cassandra: docker run --name my-cassandra -p 9042:9042 -d cassandra.
2. In cqlsh, create the keyspace and table:

   create keyspace if not exists kestra with replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
   use kestra;
   CREATE TABLE kestra.products (product_id int, product_name text, product_category text, brand text, PRIMARY KEY (product_id));
   

3. Start Redis: docker run --name my-redis -p 6379:6379 -d redis.
4. Add this flow in Kestra and let the trigger arm.
5. Push an event with redis-cli:

   LPUSH products '{"product_id": 1, "product_name": "streamline turn-key systems", "product_category": "Electronics", "brand": "gomez"}'
   

6. Watch the execution fire and the row land in kestra.products. Sample data is available in products.csv.

How to extend

• Point the trigger at a different Redis key or instance to ingest other event streams.
• Add validation or enrichment tasks before the Cassandra insert.
• Fan out to additional sinks (a warehouse, an object store, a notification) after the write.
• Swap the inline jq mapping to handle a richer payload schema.

Links

• plugin-redis documentation
• Redis orchestration with Kestra