How Acxiom Built a Multi-Tenant Data Platform on Kestra
Most orchestration deployments start as internal tools. Acxiom built something different. The company embedded Kestra as the orchestration layer at the base of its Axiom Cloud product suite, where every enterprise client runs as an isolated tenant with dedicated workers, secrets, and storage. What began as a single data ingestion product has expanded into a multi-product platform, with Kestra providing the shared foundation beneath all of it, serving 50+ enterprise clients from a single, consistent deployment model.
Declarative
dynamic & metadata-driven workflows
120+
engineers empowered
50+
customers isolated in multi-tenant environments
"Kestra was the only tool that combined true multi-tenant isolation, metadata-driven orchestration, and easy integration with our existing AWS and Databricks environments. It provided the foundation we needed to scale confidently."
Acxiom's enterprise clients handle sensitive customer data and require strict separation at every layer: storage, compute, secrets, and access controls. The internal orchestration system the team had built couldn't support true per-client isolation without substantial custom engineering for each new customer. As the client base grew, that approach became untenable.
02
Existing Scala and Python pipelines needed to run without being rewritten
Acxiom's data engineers had built pipelines in Scala and Python over years. Any new orchestration layer would need to wrap that existing code, not replace it. Tools that required migrating to a proprietary runtime or rewriting workflows in a new framework were disqualified before evaluation began. The constraint was non-negotiable: existing code runs as-is, or the tool doesn't make the cut.
03
Metadata-driven workflows at scale had no clean implementation path
Axiom Cloud is a multi-product platform. Different clients receive different data products, each with different schemas, schedules, and business rules. Hardcoding logic per client was the only path available with the existing tooling, and it didn't scale. Acxiom needed parameterized, metadata-driven workflows that could adapt to each client's configuration without rebuilding pipelines from scratch.
// The requirement
An orchestration layer that could serve as the foundation for a commercial multi-tenant SaaS platform, not just an internal tool.
What Kestra fixed
✓
Per-client tenant isolation without rebuilding the platform for each customer
Kestra's multi-tenancy model gave Acxiom a way to run each enterprise client in a dedicated environment with separate AWS VPCs, S3 storage buckets, EKS worker pools, and secrets management, all provisioned through a consistent Terraform-based process rather than custom engineering. Onboarding a new client means running the same deployment playbook, not writing new infrastructure code.
✓
YAML-first orchestration that wraps existing code without rewriting it
Kestra's declarative workflow definitions let Acxiom's engineers express pipeline logic in YAML while running the underlying Scala and Python code unchanged. "We no longer manage workflows manually, we define them clearly in YAML, deploy, and trust Kestra to run them smoothly. This dramatically improved our productivity and scalability." The orchestration layer sits on top of the existing code, not inside it.
✓
Metadata-driven parameterization across the full client base
Rather than maintaining separate pipeline definitions per client, Acxiom uses Kestra's dynamic expression language to parameterize workflows from a central metadata store. A single flow template adapts to each client's schema, schedule, and business rules at runtime, without branching into per-client codebases. "Kestra's YAML-based approach is intuitive for our engineers and powerful enough to dynamically handle metadata-driven automation at scale. It changed the way we operate."
✓
Terraform-managed deployment across every environment
Kestra's Terraform provider lets Acxiom declare, version, and deploy flows across all client environments programmatically. A change to a shared workflow propagates to every tenant through the same CI/CD pipeline, with infrastructure and orchestration managed as a single, consistent stack.
Outcomes
120+ engineers self-serving
Engineers across Acxiom's data teams define and deploy workflows without waiting on a central platform team. The declarative YAML model lowered the floor for who can contribute, and Kestra's RBAC keeps each team scoped to their own namespace.
50+ enterprise clients, fully isolated
Each of Acxiom's enterprise clients runs in its own provisioned environment, with dedicated storage, secrets, and compute. Onboarding is handled through the same Terraform deployment process every time, with no bespoke infrastructure work per customer.
Kestra as the foundation of Axiom Cloud
What started as orchestration for one data ingestion product now spans the full Axiom Cloud platform. Kestra handles pipeline orchestration across multiple products, with the same multi-tenant model scaling to support new products as they launch. "Kestra gives us a scalable foundation for innovation. We're now positioned to handle today's complexity and future advancements in AI-driven analytics and real-time orchestration."
Kestra at Acxiom
Acxiom runs a hub-and-spoke architecture on AWS. A central Kestra instance serves as the control plane, orchestrating workflows that execute across per-client spoke environments. Each spoke is a dedicated EKS cluster with its own worker pool, S3 storage, and secrets configuration, isolated at the network and IAM level from every other client.
Databricks handles compute-intensive data processing jobs. Kestra orchestrates job creation and sequencing via the Databricks plugin, while managing the surrounding pipeline logic: parameter injection from metadata, dependency resolution, error handling, and retry policies. Custom plugins extend the stack for client-specific integrations, including a purpose-built AWS Glue connector for specialized transformation workloads.
The Terraform provider governs the full deployment lifecycle. Flow definitions, namespace configurations, and environment variables are all declared in code, versioned in Git, and deployed atomically across the environment fleet. Adding a new client tenant or updating a shared workflow follows the same path: a pull request, a review, and a Terraform apply.
