Scaling Big Data Operations with Kestra

No More Legacy Constraints

Acxiom has over 50 years of experience delivering powerful customer insights, identity management, and AI-powered marketing to global brands. But as their customer base and data complexity grew, the legacy orchestration system, built in-house six years earlier, could no longer efficiently handle the increasing workload. Manual scripting, limited flexibility, and lack of multi-tenancy had become a significant bottleneck.

The company urgently needed a modern orchestration platform that could scale easily, integrate with existing data pipelines in Scala and Python, and allow dynamic, metadata-driven automation without extensive re-engineering. After evaluating multiple solutions, they chose Kestra.

Flexible, Declarative, and Dynamic Orchestration

Kestra’s declarative, YAML-based workflows immediately simplified the definition and management of complex Big Data jobs. By defining pipelines as clear YAML configurations, Acxiom’s engineers could dynamically chain metadata, automate data transformations, and integrate with their existing Scala-based infrastructure. No more complicated scripting or code-heavy customizations.

Kestra’s API-first architecture simplified integration at Acxiom, allowing orchestration directly through their existing toolsets and processes. Thanks to this flexible approach, Acxiom integrated Kestra smoothly into their current DevOps practices, leveraging GitOps, version control, and automated CI/CD pipelines without relying on the UI in production environments.

"Kestra’s YAML-based approach is intuitive for our engineers and powerful enough to dynamically handle metadata-driven automation at scale. It changed the way we operate."

Operational Impact at Acxiom

With Kestra, Acxiom rapidly transitioned from legacy workflows to dynamic, automated orchestration:

Optimized Databricks Management:

Kestra enabled automated orchestration of Databricks clusters—from job submission and scaling resources dynamically, to efficient real-time workload management. Operational costs decreased, and performance improved significantly.

Multi-Tenant Isolation:

Kestra’s built-in multi-tenant architecture empowered Acxiom to run isolated workflows per customer within dedicated AWS Virtual Private Clouds (VPCs), each with separate S3 storage and EKS deployments. This met stringent security and compliance standards seamlessly.

Enterprise-Grade Reliability and Monitoring:

Acxiom gained full end-to-end visibility, observability, and governance of their workflows through Kestra’s built-in features, including auto-scaling on AWS, zero-downtime blue-green deployments, role-based access control (RBAC), and comprehensive audit logging.

"Kestra was the only tool that combined true multi-tenant isolation, metadata-driven orchestration, and easy integration with our existing AWS and Databricks environments. It provided the foundation we needed to scale confidently."

Scaling with Confidence and Visibility

Since deploying Kestra, Acxiom has experienced significant improvements across their data platform:

• 120+ engineers empowered to independently design, deploy, and manage workflows more efficiently.
• 50+ customers served securely and seamlessly with true multi-tenant orchestration.
• Significant reduction in manual scripting, troubleshooting, and orchestration overhead.

Now, workflows are easily defined and quickly deployed via Terraform, enabling rapid, reliable scaling and precise management of resources. Engineering teams spend less time on orchestration plumbing, dedicating more focus to innovation and strategic projects.

"We no longer manage workflows manually, we define them clearly in YAML, deploy, and trust Kestra to run them smoothly. This dramatically improved our productivity and scalability."

Future-Proofing Data Operations

With Kestra as the core orchestration backbone, Acxiom continues to extend automation coverage and refine its data operations. Plans include deeper integration of advanced machine learning workflows, real-time AI model execution, predictive analytics, and intelligent decision-making leveraging Kestra’s event-driven capabilities.

"Kestra gives us a scalable foundation for innovation. We’re now positioned to handle today’s complexity and future advancements in AI-driven analytics and real-time orchestration."