How Riverside Connected dbt Cloud, Snowflake, and Hightouch Into a Single Dependency-Aware Pipeline
Riverside is a podcast recording and editing platform used by creators and media teams worldwide. Their analytics engineering team runs the company's internal data operations — ELT ingestion into Snowflake, sequential dbt Cloud transformation jobs, Metaplane data quality checks, and Hightouch reverse ETL to downstream platforms. Six tools in total, all with dependencies on each other, all previously running on isolated time-based schedules with no actual handoff between them. The team evaluated Astronomer, Dagster, Mage, and Orchestra. All four were rejected. Airflow was too complex for a team of SQL and YAML practitioners. Dagster required too steep a learning curve. Orchestra abstracted too much away. They chose Kestra Cloud and connected their full analytics stack without rewriting a single pipeline.
6+
data tools orchestrated
4
orchestrators evaluated
Astronomer, Dagster, Mage, and Orchestra all rejected
0
pipeline rewrites
existing scripts and tools stay in place
2
engineers maintaining all production flows
no DevOps required
"Simply powerful yet simple enough."
The problem
Six tools with real dependencies on each other — coordinated only by timers.
Riverside's analytics engineering team runs the company's internal data operations. Every business insight moves through a pipeline — ELT ingestion into Snowflake, sequential dbt Cloud transformation jobs, Metaplane data quality checks, and Hightouch reverse ETL to downstream platforms. Six tools, all with real dependencies on each other, all previously running on isolated schedules with no actual handoff between them.
01
Pipelines ran on timers, not dependencies
Each tool assumed the previous step had finished. If ELT was still running when dbt's scheduled slot arrived, the transformation would execute on incomplete data. No alert fired. The cost became visible when the CEO flagged a number that looked off — every tool had run on schedule, but the data was wrong because the sequence had broken silently. "It undermines credibility."
02
Python-first orchestrators required expertise the team didn't have
The analytics engineering team writes SQL, YAML, and Python transformation logic — not platform engineering. Astronomer and Dagster both required adopting a Python-centric execution model: decorators, framework abstractions, proprietary asset definitions. An orchestrator that required learning a new programming model was not viable for a team whose job is data transformation, not infrastructure ownership.
03
The final choice came down to abstraction vs. control
Orchestra and Kestra were the final two. Orchestra hides the YAML entirely — appealing, but that meant less control over flow structure and fewer options for the custom connectors the team needed for tools like Metaplane. Kestra's YAML gave the team direct control over sequencing, conditional logic, and custom API calls. After the trial: "Simply powerful yet simple enough."
// The requirement
Tools that could talk to each other — not just run on timers.
What Kestra fixed
Developer-experience arguments won the eval. Dependency-aware pipelines won the team.
✓
Declarative orchestration at the right level of abstraction
Kestra sits on top of Riverside's existing tools. Nothing was rewritten. Custom Python scripts stay in their own repository — Kestra calls them. dbt Cloud jobs run as sequential steps, each dependent on the last. Analytics engineers can add a step to an existing flow without understanding how Kestra executes it underneath. "You give flexibility, but you abstract that with a declarative approach in which you say, okay, I want this to happen and it will happen."
✓
Data quality gates embedded between every pipeline stage
Metaplane checks run between dbt stages and halt the pipeline if a check fails, sending a Slack alert before bad data reaches stakeholders. The CEO had previously flagged a number that looked off because the sequence had broken silently. That's why these gates exist at every stage.
✓
Event-driven triggers for the Snowflake handoff
Some of Riverside's ELT tools run continuously and don't emit a clean completion event. Kestra polls a Snowflake table for the latest run status, confirms success, and only then kicks off the dbt pipeline — no timers, no guesswork.
✓
Managed cloud, no infrastructure required
"We don't have the technical knowledge nor the help from our teams to set up the infra. We just need something available to use right from the get-go." Self-hosted was never on the table. Kestra Cloud gave them a production-ready instance without a DevOps engagement. Two engineers maintain all production flows.
Outcomes
6+ tools in sync
ELT ingestion, sequential dbt Cloud jobs, Metaplane data quality gates, and Hightouch reverse ETL now run in a dependency-aware sequence instead of on isolated timers.
Quality gates at every stage
Metaplane checks between every dbt stage halt the pipeline on failure and fire a Slack alert. Bad data no longer reaches stakeholders silently.
2 engineers own it all
Hernán and one other engineer maintain all production flows. Team members without deep Kestra knowledge can read a flow and add a step. "They don't need to know a lot to read an extra flow."
Kestra in the Riverside analytics stack
Riverside runs Kestra Cloud on GCP. Custom Python scripts and Fivetran connectors handle ELT ingestion from external sources. Kestra triggers sequential dbt Cloud jobs via the API connector. Metaplane checks run between dbt stages via a custom HTTP connector, halting the pipeline on failure. Hightouch is triggered after dbt completes to sync results to downstream platforms. Slack handles alerting throughout — pipeline failures, data quality alerts, and development-mode suppression via tag-based conditions.
