Introduction
Ntico, a French consulting and software firm, specializes in data-driven applications and complex project management. One of their offerings is Locxia, a data platform aimed at providing businesses and local authorities with comprehensive insights into territories and their inhabitants. When faced with the challenge of managing enormous volumes of geospatial data, Ntico turned to Kestra as its orchestration platform.
Technology Stack
Ntico's technology stack is a blend of specialized tools, each serving a unique function but integrated to work in harmony:
- AWS S3: The primary data lake where both raw and processed data are stored.
- AWS Athena: Used for ad-hoc querying and data analysis.
- PostgreSQL: Manages transactional and relational data.
- Kestra: The orchestrator that ensures data flow between these components, handling workflow dependencies, error management, and scheduling.
The Challenge
Ntico faced several hurdles in developing Locxia:
- Volume: The platform had to manage terabytes of geospatial data.
- Diversity of Data Sources: Data was being ingested from various sources including APIs, open-data platforms, and raw satellite images.
- Scalability and Cost: The solution had to be scalable and cost-effective to handle the increasing data volume.
Why Kestra?
Ntico chose Kestra for several key reasons:
- Advanced Data Ingestion and Processing: Kestra could handle diverse data sources and efficiently push this data into an AWS S3 bucket.
- User Experience: Kestra's user interface provided a visual representation of task dependencies, aiding in effective workflow management.
- Customization: Kestra's support for Docker allowed Ntico to bring their own business logic and technical requirements into the platform.
Outcome
With Kestra, Ntico successfully tackled their challenges head-on:
- Efficient Data Management: Kestra coordinated the data ingestion processes, effectively handling data from diverse sources.
- Scalability: Kestra's capabilities proved to be scalable, meeting the growing needs of Locxia.
- User Empowerment: The user-friendly interface and customization options allowed Ntico to design and build tailored data pipelines, meeting their specific requirements.
Conclusion
The collaboration between Ntico and Kestra has been pivotal in the development of Locxia. Kestra's advanced capabilities, scalability, and customization options have played a crucial role in enabling Ntico to deliver a robust, reliable, and efficient data platform.
If you want to learn more about the journey of Loxcia and Ntico with Kestra checkout our blog post.
