Leroy Merlin France

Leroy Merlin France, enabling a datamesh architecture and 900% increase in data production with Kestra

A screenshot of the user interface of Kestra's application

Introduction

Leroy Merlin France, a global retail market leader, employing over 24,000 people was at a crucial crossroads in its digital transformation journey. Their existing data architecture was incompatible with their evolution toward a cloud based infrastructure.

they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a datamesh architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics.

Technology stack before Kestra

  1. Database: Teradata was initially used, but it posed challenges in scalability and flexibility.
  2. Data Integration: Talend was in use but proved costly and less agile.
  3. Scheduling and Operations: Older tools like Dollar U and Automic Workload Automation were employed but lacked modern, interconnected workflow 

Challenges Faced

The technology stack was bogged down by several bottlenecks:

  • Infrastructure Bottleneck: The need for rapid migration to a serverless cloud architecture.
  • Data Pipeline Bottleneck: Re-architecting was required to move from a centralized data team to a decentralized data integration led directly by the product team.
  • Delivery and Automation Bottleneck: The adoption of CI/CD and DataOps principles were essential for improving data operations

Technology stack with Kestra

  • Cloud Platform (Google Cloud): The transition to Google Cloud from on-premises was essential. Google Cloud's serverless architecture complemented Kestra's capabilities.
  • Database (BigQuery): Used for storing massive data lakes and quick querying. Kestra workflows would push or pull data as necessary, making BigQuery more than just a storage solution.
  • CI/CD Tools (Terraform, GitHub Actions): These tools integrated seamlessly with Kestra, automating many of the deployment and update processes, making DataOps a reality.
  • Data Storage (Google Cloud Storage Bucket): Temporary data storage needs were met through Google Cloud Storage, which Kestra could interact with to stage or fetch data.
  • Data Orchestration (Kestra): Kestra integrated well with BigQuery and GCS for streamlined data transfers and transformations.

Results and Outcomes

  • Surge in Data Production: A 900% increase in data production was observed.
  • Scalability: From less than half a million tasks to over 5 million tasks monthly.
  • Cost and Time Efficiency: Significant improvements in data processing times and costs.

The adoption of Kestra not only solved the existing challenges but also laid the foundation for a datamesh architecture. Within 18 months, user adoption skyrocketed by more than 900%, with users executing millions of tasks monthly, proving the vast capabilities of Kestra.

Getting started with Kestra

Start building with Kestra — Automate Everything Everywhere All at Once.

Read the docsGet started!