Introduction
Leroy Merlin France, a global retail market leader, employing over 24,000 people was at a crucial crossroads in its digital transformation journey. Their existing data architecture was incompatible with their evolution toward a cloud based infrastructure.
they discovered Kestra, a tool that not only fulfilled the initial requirements but also unlocked the potential for a datamesh architecture, enabling several hundred data practitioners to collaboratively and securely produce high-quality data analytics.
Technology stack before Kestra
- Database: Teradata was initially used, but it posed challenges in scalability and flexibility.
- Data Integration: Talend was in use but proved costly and less agile.
- Scheduling and Operations: Older tools like Dollar U and Automic Workload Automation were employed but lacked modern, interconnected workflow
Challenges Faced
The technology stack was bogged down by several bottlenecks:
- Infrastructure Bottleneck: The need for rapid migration to a serverless cloud architecture.
- Data Pipeline Bottleneck: Re-architecting was required to move from a centralized data team to a decentralized data integration led directly by the product team.
- Delivery and Automation Bottleneck: The adoption of CI/CD and DataOps principles were essential for improving data operations
Technology stack with Kestra
- Cloud Platform (Google Cloud): The transition to Google Cloud from on-premises was essential. Google Cloud's serverless architecture complemented Kestra's capabilities.
- Database (BigQuery): Used for storing massive data lakes and quick querying. Kestra workflows would push or pull data as necessary, making BigQuery more than just a storage solution.
- CI/CD Tools (Terraform, GitHub Actions): These tools integrated seamlessly with Kestra, automating many of the deployment and update processes, making DataOps a reality.
- Data Storage (Google Cloud Storage Bucket): Temporary data storage needs were met through Google Cloud Storage, which Kestra could interact with to stage or fetch data.
- Data Orchestration (Kestra): Kestra integrated well with BigQuery and GCS for streamlined data transfers and transformations.
Results and Outcomes
- Surge in Data Production: A 900% increase in data production was observed.
- Scalability: From less than half a million tasks to over 5 million tasks monthly.
- Cost and Time Efficiency: Significant improvements in data processing times and costs.
The adoption of Kestra not only solved the existing challenges but also laid the foundation for a datamesh architecture. Within 18 months, user adoption skyrocketed by more than 900%, with users executing millions of tasks monthly, proving the vast capabilities of Kestra.
