Leroy Merlin France

Introduction

Leroy Merlin France, a global retail market leader, employing over 24,000 people was at a crucial crossroads in its digital transformation journey. Their existing data architecture was incompatible with their evolution toward a cloud based infrastructure.

they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a datamesh architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics.

Technology stack before Kestra

1. Database: Teradata was initially used, but it posed challenges in scalability and flexibility.
2. Data Integration: Talend was in use but proved costly and less agile.
3. Scheduling and Operations: Older tools like Dollar U and Automic Workload Automation were employed but lacked modern, interconnected workflow 

Challenges Faced

The technology stack was bogged down by several bottlenecks:

• Infrastructure Bottleneck: The need for rapid migration to a serverless cloud architecture.
• Data Pipeline Bottleneck: Re-architecting was required to move from a centralized data team to a decentralized data integration led directly by the product team.
• Delivery and Automation Bottleneck: The adoption of CI/CD and DataOps principles were essential for improving data operations

Technology stack with Kestra

• Cloud Platform (Google Cloud): The transition to Google Cloud from on-premises was essential. Google Cloud's serverless architecture complemented Kestra's capabilities.
• Database (BigQuery): Used for storing massive data lakes and quick querying. Kestra workflows would push or pull data as necessary, making BigQuery more than just a storage solution.
• CI/CD Tools (Terraform, GitHub Actions): These tools integrated seamlessly with Kestra, automating many of the deployment and update processes, making DataOps a reality.
• Data Storage (Google Cloud Storage Bucket): Temporary data storage needs were met through Google Cloud Storage, which Kestra could interact with to stage or fetch data.
• Data Orchestration (Kestra): Kestra integrated well with BigQuery and GCS for streamlined data transfers and transformations.

Initial Challenges and Airflow's Limitations

The transition revealed critical challenges, particularly with their initial choice of Apache Airflow for data orchestration. Airflow's complexity, reliability issues, and the lack of support for inter-team flow triggering presented substantial hurdles. Moreover, the increased processing times and resource demands, coupled with escalating costs, prompted Leroy Merlin France to seek an alternative solution.

Kestra as the Solution

Kestra emerged as a robust alternative, addressing Airflow's shortcomings with its simplicity, reliability, and efficiency. Its declarative nature and event-driven architecture offered Leroy Merlin France the agility and scalability needed for its growing data operations. Kestra's introduction streamlined workflow management and also paved the way for implementing a Data Mesh Architecture, empowering teams to produce high-quality analytics collaboratively.

Implementing a Data Mesh Architecture with Kestra

The adoption of Kestra facilitated a shift towards a Data Mesh Architecture, where various data products and teams could independently manage their data domains. This decentralized approach allowed for a more scalable and responsive data infrastructure, supporting the rapid growth of Leroy Merlin France's data production by 900% over two years.

Overcoming Airflow's Limitations with Kestra

Kestra's unique architecture and intuitive design simplified the complexity associated with managing data workflows, offering a more reliable and cost-effective orchestration solution. Its ability to efficiently handle dataflows significantly reduced processing times, aligning with Leroy Merlin France's operational requirements.

Use Cases Highlighting Kestra's Impact at Leroy Merlin France

In its transformation journey, Leroy Merlin France leveraged Kestra for specific operational challenges, directly contributing to the scaling of its data analysis and science capabilities. By automating the creation of SQL pipelines, Kestra enabled data analysts and scientists at Leroy Merlin France to shift their focus toward solving core business issues, significantly reducing the time previously dedicated to data preparation and wrangling.

Furthermore, the integration of Kestra with DevOps tools, such as Terraform and GitHub Action, simplified the deployment and management of data workflows. This streamlined approach allowed for more efficient adjustments and scaling of Leroy Merlin France's data infrastructure, ensuring that their data management capabilities could evolve alongside their growing business needs.

Another pivotal use case was the collection and integration of customer feedback into the data warehouse. Utilizing Kestra's workflows, Leroy Merlin France was able to orchestrate the process of gathering, processing, and storing customer feedback efficiently.

Conclusion

Leroy Merlin France's journey with Kestra marks a significant milestone in their data management transformation. By addressing the limitations of previous tools and embracing a Data Mesh Architecture, the company has laid a robust foundation for future growth and innovation in data analytics. Kestra has streamlined Leroy Merlin France's data processes and empowered their teams to collaborate more effectively.