Introduction
Gorgias, a company focused on enhancing the user experiences of online merchants and shoppers, has been at the forefront of leveraging technology to achieve its goals. To build and manage its data infrastructure, Gorgias has turned to Infrastructure as Code (IaC) as a key enabler, using an array of modern tools such as Kestra, Airbyte, dbt, and Hightouch.
Technology Stack
Gorgias employs a robust technology stack to manage and automate its data operations:
- Airbyte: Manages data integration from various sources.
- dbt (Data Build Tool): Transforms raw data into analytics-ready models.
- Hightouch: Syncs transformed data back to applications, enabling data-driven operations.
- Kestra: Handles workflow orchestration and ensures data tasks run smoothly.
The Challenge
Prior to this, Gorgias faced several challenges with their data operations, mainly related to complex orchestration, logging, and scheduling. The tools they were using had limitations in handling retries, triggering events, and managing dependencies, leading to inefficient workflows and time-consuming manual intervention.
Why Kestra?
Gorgias chose Kestra for its:
- Native Integrations: Seamless compatibility with tools like Airbyte, BigQuery, PostgreSQL, dbt, GitHub, Slack, and Hightouch.
- Rich UI: Enables easy task definition and flow management.
- YAML-Based Flow: Aligns with their Infrastructure as Code approach.
- Scalability: Designed to handle millions of flows in parallel.
The Outcome
With Kestra, Gorgias was able to:
- Simplify Orchestration: Triggering Airbyte sync followed by dbt transformations has become efficient.
- Automate Retries: Particularly for dbt jobs, managing resource exhaustion issues has become more automated.
- Streamline Scheduling: Hightouch sync now occurs seamlessly at the end of dbt jobs.
What's Good About Kestra
Kestra's Infrastructure as Code (IaC) alignment ensures consistency, speed, and efficiency in deployment and management. The modular design and versioning make it easier to adapt and scale the infrastructure as needed.
What's Next
Gorgias plans to empower more team members to use Terraform and YAML for defining their flows. They are also looking to implement proper CI/CD with sandbox environments, and are considering replacing some business automation tools like Zapier with Kestra, which offers low-code editing capabilities.
