Displayce

Displayce, a subsidiary of JCDecaux, serves as the intermediary between JCDecaux, agencies, and advertisers. Their role is critical in advising clients on optimal assets and marketing campaigns. Faced with the challenge of scaling their data operations and lacking a unified system for monitoring and orchestration, Displayce sought a robust solution to streamline their processes. Their search led them to Kestra.

Technology Stack

Displayce employs a multifaceted technology stack, each component serving a distinct purpose, yet all integrated to function as a cohesive unit.

• AWS S3: Utilized as the primary data lake, AWS S3 stores raw and processed data, acting as the foundation for all data operations.
• AWS Athena: Athena is employed for ad-hoc querying and analysis. It enables Displayce to pull insights directly from their S3 data lake without requiring a separate data warehouse.
• Clickhouse: This columnar database serves as their analytical data store, optimized for query performance. It allows them to run complex analytics queries quickly and efficiently.
• PostgreSQL: Used for transactional data and to maintain relational data structures, PostgreSQL supports application-level data requirements.
• dbt: As their transformation engine, dbt is employed to clean, enrich, and prepare data for analytics. It provides a robust layer of transformation logic that feeds into Clickhouse.

Kestra: Kestra ensure seamless data flow between these components. It manages workflow dependencies, error handling, and scheduling, thus providing the robust backbone needed for their operations.

Overcoming Growth-related Challenges

As Displayce’s operations expanded, they encountered significant hurdles. Initially reliant on CLI-based tools for data ingestion, they found these methods increasingly unsustainable. Moreover, the absence of a centralized orchestration and monitoring system became a pressing issue, complicating their ongoing migration efforts from DigDag and Nomad, which added layers of complexity to their operational upgrade.

Why Kestra Became the Chosen Solution

Kestra emerged as the preferred solution for several reasons. Its comprehensive plugin ecosystem allowed for effortless integration with Displayce’s existing tools, simplifying data flows significantly. The integration enabled Displayce to create a reliable data transformation pipeline with dbt, ensuring data was primed for analytics in Clickhouse. This strategic choice allowed Displayce to not only maintain but expand their workflow operations, now overseeing an impressive array of active workflows with increased efficiency and control.

The Impact of Integrating Kestra

With the adoption of Kestra, Displayce efficiently addressed their challenges, achieving scalability and establishing a centralized platform for nuanced monitoring and orchestration. This shift facilitated a smooth transition from legacy systems, enhancing operational efficiency and elevating the quality of service provided to their clients. Kestra’s role in Displayce’s data ecosystem marked a pivotal step in their technological advancement, optimizing their data operations to support their critical advisory role in the advertising industry.

Moving Forward with Optimized Data Management

Kestra’s integration into Displayce’s technological framework has transformed their approach to data management, enabling scalable, efficient, and controlled operations. This strategic update has streamlined internal processes but also amplified their capability to deliver scalable data pipelines.