How One Engineer Manages 50+ Production Pipelines for 2,800 Automotive Clients
Foundation Data serves over 2,800 automotive dealerships across the United States. One engineer runs the entire data stack. That ratio became possible because of two decisions made early in the company's life as an independent entity: commit to Infrastructure as Code from day one, and find an orchestrator that could run Go. Both pointed to Kestra.
50+
workflows in production
10x
faster to build
>90%
toolchain cost reduction
99%
pipeline success rate
"Kestra completely transformed our operational workflows. We now deploy new pipelines rapidly, easily manage our integrations, and have vastly improved our productivity and cost efficiency."
The problem
01
Python-only orchestration couldn't run the language the workload required
Foundation Data's OEM data extraction jobs query dealership APIs that are rate-limited. Python's threading model capped concurrent requests at 3. Go's concurrency model supports 50 to 60 simultaneous requests. For extraction jobs that were taking 2 to 3 hours, that difference meant finishing in under an hour or missing SLAs entirely. The orchestrator needed to run Go natively, not as a workaround but as a first-class execution target. "We've had problems in the past where like it's released and it's broken stuff, and we had to wait for like a day or two to get it fixed. Our legacy tools just couldn't scale and adapt to our operational needs." — Jack Perry
02
A three-tool stack with mounting costs and no path to Infrastructure as Code
As Foundation Data spun out as an independent company in mid-2025, the cost structure needed to work at the new entity's scale. The existing toolchain carried significant fixed costs: SuperMetrics for data fetching, dbt Cloud for transformations, and Prefect workspace environments on top. The orchestration layer had no Terraform support, which blocked Foundation Data's commitment to deploy everything as versioned, rollback-able infrastructure. The toolchain wasn't just expensive; it was architecturally incompatible with how the team wanted to build.
03
Non-technical users needed to trigger complex workflows without engineering help
Foundation Data's measurement team (four or five analysts who monitor Google Analytics across 2,800 dealership websites) needed to trigger workflows on demand: updating tags, running audits, syncing changes across hundreds of dealer sites simultaneously. These workflows involve 30 to 40 conditional inputs and six or seven underlying subflows. The team couldn't route every request through the engineering queue, and they couldn't give analysts access to raw pipeline tooling.
// The requirement
An orchestrator that could run any language, deploy via Terraform, and give non-engineers controlled access to complex workflows, without requiring a second engineer to manage it.
What Kestra fixed
✓
Language-agnostic task runners: Go, Python, Bash, SQL in the same pipeline
Kestra's task runners execute code in isolated containers, without forcing a single runtime. Foundation Data's OEM extraction jobs now run as Go containers. dbt transformations run as dbt Core. Python and Bash tasks run alongside them in the same flow. Extraction time dropped from 2 to 3 hours to under an hour. "Kestra's architecture and ease of use immediately appealed to us. The YAML-based approach dramatically simplified our workflow management, letting us deploy faster and scale seamlessly." — Jack Perry
✓
Full toolchain consolidation: SuperMetrics, Prefect, and dbt Cloud all replaced
SuperMetrics was decommissioned. dbt Cloud was replaced with dbt Core, orchestrated directly by Kestra. Prefect workspace fees were eliminated. The monthly toolchain cost fell by more than 90%, leaving almost entirely BigQuery compute. "The cost savings with Kestra have been tremendous. Moving away from Supermetrics immediately slashed our monthly spend, while Kestra's enhanced concurrency management further optimized our costs." — Jack Perry
✓
Full infrastructure managed through Terraform, including Kestra itself
Foundation Data's entire Kestra environment (namespaces, secrets, users, flows, and Apps) is declared in Terraform and deployed from Git. That was a hard requirement from day one of the company's independent existence. Kestra was the only orchestrator in the evaluation that supported it. "Kestra empowered us with a real DevOps-centric approach. Our deployment pipeline is streamlined, and our team can scale rapidly without ever sacrificing stability." — Jack Perry
✓
Kestra Apps: non-engineers triggering complex workflows with guardrails
The measurement team uses a Kestra App built by Jack to manage Google Analytics across all 2,800 dealership websites. The App presents 30 to 40 conditional inputs, routes users through the right options, and handles six or seven underlying subflows automatically. Concurrency controls cap execution at Google Tag Manager's rate limit of 200 requests per minute, queuing simultaneous runs rather than letting them collide. "Deploying workflows went from a cumbersome task to something we handle effortlessly in minutes. Our engineers are now free to focus on innovation rather than troubleshooting manual scripts." — Jack Perry
Outcomes
>90% toolchain cost reduction
Foundation Data replaced a three-tool stack with a single orchestration platform. Consolidating onto Kestra cut toolchain costs by more than 90%, leaving almost entirely BigQuery compute. That shift funded the operational model that lets one engineer run the whole thing.
40% client growth at 10% cloud cost increase
Foundation Data grew from 2,000 dealership clients to 2,800 between 2024 and 2025. Their cloud bill grew roughly 10%. The architecture scaled horizontally without requiring proportional infrastructure spend or additional headcount. "We went from 2,000 to 2,800 dealers — 40% growth — and our cloud bill went up maybe 10%. I don't think people realize how much Jack has made our systems efficient." — Mike Heidner, SVP Analytics
99% pipeline success rate
Under the previous orchestrator, pipeline failures came roughly every other day. Kestra has had one failure in fourteen months. "Switching to a dedicated Kestra cloud instance was seamless and immediately improved our operational stability. The performance gains and additional control we received have been invaluable." — Jack Perry
One engineer, 50+ production workflows
Jack Perry manages the full data stack as Foundation Data's sole data engineer. Three outsourced data contractors were let go. Mike Heidner, Foundation Data's SVP of Analytics, monitors pipeline health and executes flows himself from the Kestra UI daily. "Today I approved five flows in Kestra just sitting there. Two clicks of a button. I feel like I'm more aware of what's going on with our data pipelines than I ever was before." — Mike Heidner, SVP Analytics. "Kestra has fundamentally changed the way we manage our data workflows. We now deploy pipelines faster, manage integrations, and scale confidently. It's essential for our continued growth and success." — Jack Perry
Kestra at Foundation Data
Foundation Data runs on Kestra Cloud (GCP US Central, dedicated instance). Jack Perry manages the environment entirely through Terraform: namespaces, secrets, user permissions, flow definitions, and Kestra Apps are all declared in code and deployed from Git. Adding a new OEM data source means writing Go code for the extraction logic and a Kestra flow definition for the orchestration; the infrastructure is a merge.
OEM extraction jobs run in containerized Go workers that query dealership APIs with 50 to 60 concurrent requests. The previous Python-based approach was limited to 3 concurrent requests, turning a rate-limited API into a multi-hour bottleneck. dbt Core runs directly inside Kestra, replacing dbt Cloud without changing any transformation logic. BigQuery is the analytics target; Looker dashboards sit on top for client-facing reporting.
The measurement team's workflow (monitoring and updating Google Analytics across 2,800 dealership websites) runs through a Kestra App. Analysts trigger updates on demand through a guided interface. Concurrency controls and API rate-limit queuing are declared in the flow definition, not written into application code, which means they're visible, reviewable, and consistent regardless of who triggers the workflow.
Kestra CloudGoPythonBigQuerydbt CoreTerraformLookerGoogle Tag Manager
Experience Kestra Today
What would change if one engineer could orchestrate 50+ pipelines, serve 2,800 clients, and still have time to innovate?
