CI/CD for your Kestra Workflows
Embrace software engineering practices for versioning, deploying, and maintaining data workflows
Get started
What is CI/CD?
CI/CD stands for Continuous Integration and Continuous Deployment, software engineering practices that involve regularly merging code changes (Continuous Integration) and automated deployment to production (Continuous Deployment). CI/CD practices make the development process more efficient, less error-prone, and better aligned with business needs.Build your Workflows as Code
Kestra allows building workflows as code. This approach facilitates seamless integration with existing CI/CD processes, Infrastructure as Code tools like Terraform, and version control systems like Git.
Implementing CI/CD for Kestra Flows
You can version-control your Kestra workflows in a Git repository, easily track changes, collaborate with other team members, and roll back to previous versions if necessary.
The CI/CD pipeline automatically validates and deploys the updated workflows to the Kestra environment whenever changes are committed and pushed to a given Git branch.
From Development to Production
Creating data workflows often starts locally and gradually moves towards production. It involves designing data pipelines, setting up tasks, dependencies, and verifying the code's functionality.
.
The typical workflow of moving a data pipeline from development to production starts with a pull request. Once approved, the changes get deployed to the production. .
.
This streamlined approach ensures that the production environment remains stable and secure while allowing for rapid adaptation to changing business needs and requirements.
Manage DataOps Lifecycle with Kestra and Terraform
Combining Kestra and Terraform provides a reliable foundation for managing the data engineering lifecycle while promoting collaboration between engineers and business users.
The combination of Infrastructure as Code and Workflow as Code enables organizations to embrace engineering best practices, balancing fast feedback loops during development with the benefits of infrastructure as code for production environments.
Fast iteration cycles
Kestra enables both data teams and business stakeholders to participate in workflow creation using a simple declarative configuration.
Kestra UI provides an embedded code editor, versioning, auto-completion, and a topology view of workflow dependencies, simplifying the development process for all stakeholders. This way, Kestra facilitates collaboration and fast iteration cycles.
Apply DevOps best practices in your data engineering lifecycle
Kestra provides a comprehensive solution to streamline the data engineering lifecycle by offering collaborative development and infrastructure as code capabilities. Through its user-friendly interface and integration with tools like Terraform, Kestra enables organizations to develop and deploy data workflows efficiently while maintaining the stability and security of production environments.
Key Features
Workflows as Code
Define and manage data workflows using YAML configuration, which opens data workflows to more people in your organization.
Versioning
Kestra workflows are automatically versioned, helping track changes and rollback to previous versions if needed.
CI/CD Integration
Kestra integrates with Terraform, GitLab, and GitHub Actions to automate the deployment of data workflows to production environments.
Software Engineering Best Practices
Kestra embraces software engineering best practices, promoting high-quality and reliable data pipelines.