Introduction
Displayce, a subsidiary of JCDecaux, serves as the intermediary between JCDecaux, agencies, and advertisers. Their role is critical in advising clients on optimal assets and marketing campaigns. Faced with the challenge of scaling their data operations and lacking a unified system for monitoring and orchestration, Displayce sought a robust solution to streamline their processes. Their search led them to Kestra.
Technology Stack
Displayce employs a multifaceted technology stack, each component serving a distinct purpose, yet all integrated to function as a cohesive unit.
- AWS S3: Utilized as the primary data lake, AWS S3 stores raw and processed data, acting as the foundation for all data operations.
- AWS Athena: Athena is employed for ad-hoc querying and analysis. It enables Displayce to pull insights directly from their S3 data lake without requiring a separate data warehouse.
- Clickhouse: This columnar database serves as their analytical data store, optimized for query performance. It allows them to run complex analytics queries quickly and efficiently.
- PostgreSQL: Used for transactional data and to maintain relational data structures, PostgreSQL supports application-level data requirements.
- dbt: As their transformation engine, dbt is employed to clean, enrich, and prepare data for analytics. It provides a robust layer of transformation logic that feeds into Clickhouse.
Kestra: Kestra ensure seamless data flow between these components. It manages workflow dependencies, error handling, and scheduling, thus providing the robust backbone needed for their operations.
The Challenge
Displayce faced several significant hurdles in their journey:
- Scalability: They began with CLI-based tools for data ingestion, which proved unscalable as their operations grew.
- Monitoring and Orchestration: The absence of a centralized system for monitoring and orchestration was glaring.
- Migration: Adding to the complexity was their ongoing migration from DigDag (a Java-based scheduler) and Nomad (a container orchestration platform).
Why Kestra?
After evaluation, Displayce chose Kestra for several compelling reasons:
- Robust Plugin Ecosystem: Kestra's rich suite of plugins facilitated easy integrations with AWS S3, Athena, and Clickhouse.
- Transformation: Leveraging dbt, they created a reliable data transformation layer that could feed refined data into Clickhouse for analytics.
- Expanding Workflows: Since adopting Kestra, Displayce's operations have grown exponentially; they now maintain over 50 active workflows.
Outcome
With Kestra, Displayce has efficiently addressed their challenges, achieving not just scalability but also setting up a centralized platform for monitoring and orchestration. They've transitioned smoothly from their legacy systems of DigDag and Nomad, ultimately streamlining their operations and enhancing the quality of service to their clients.
