Fast, Scalable, and Declarative Data Management
Create and Deploy all kinds of Data Pipelines with Ease and Speed with Kestra: Your All-in-One Solution for Orchestrating Workflows
Add new workflows in minutes. No ops required.
Finally, no dependency conflicts. It just works!
API-first design with declarative workflow definition
- Strong typing and autocompletion: iterate quickly when adding new flows, triggers and tasks. If something is incorrect in your workflow definition, you will know in advance thanks to the built-in syntax validation.
- Support for data-mesh: thanks to the simple YAML interface, built-in documentation and blueprints, domain experts can build their own workflows, too.
- Intuitive UI built not only for engineers: when you add workflows directly from the UI, the live-updating topology view helps you see dependencies between processes, systems and tasks.
Iterate faster without the overhead of CI/CD
- Less time spent babysitting orchestration code: Seamlessly configure your tasks to run sequentially, in parallel or only based on a specific condition or presence of events without having to navigate convoluted orchestration code bases.
- Intuitive dependency management: Easily manage task dependencies — defining proper execution order is as simple as adding a list of tasks in the order you want them to be executed.
- Built-in documentation and blueprints: No need to navigate bulky documentation to figure out the right task or trigger properties — they are available right next to your code editor.
Flexible Event-Driven Workflow Execution
- Time-based scheduling: Workflows run automatically at specific times, such as daily, weekly, or monthly without the need for manual intervention or inspecting crontabs.
- Event-based orchestration: Automate workflow executions based on real-time events. Trigger workflows to run automatically in response to events, such as the arrival of new data in a database, file detection in your cloud storage bucket, new message in a message queue, or the completion of other workflows.
Execute and Monitor in Real Time
- Real-time Workflow Monitoring: Progress tracking of workflows and individual tasks in real-time. Troubleshoot failure by replaying failed tasks or doing a full-text search of the workflow execution logs.
- Actionable Insights: you can build custom dashboards to track workflow performance using built-in outputs and metrics. Thanks to a standardized format that Kestra uses, all your Kestra metrics can be visualized in Grafana or Kibana.
Simplicity scales, complexity fails
Save time and development costs.
Less time spent writing boilerplate orchestration code, more time for what matters to your business
Deploy and Run Anywhere
From on-premise installations to cloud-based services, containers to clusters, Kestra can be deployed anywhere.
Seamless Integrations
Connect to various data sources and cloud services like AWS, GCP, and Microsoft Azure.
Scale to Millions Executions
Kestra has been built with leading cloud-native technologies and a strong architecture, enabling scaling to millions of executions without hassle.
Simple Integration
Customize and extend Kestra's integrations with custom plugins, tailored to your specifics needs. Integrate internal systems, third-party APIs, cloud services or databases using simple and scalable integrations.
See open-source pluginsCreate a custom pluginSeamless Workflow Design and Management
Create your Flows, Avoid the Headache
Simplicity
Easily create, configure, and modify flows with Kestra's YAML format and user interface.
Helpful
Error spotting directly in the editor before any runtime error.
Allow other programming languages
Run any scripting language like Python, Bash, and Node.js using Kestra.
Use workflows as code
Deploy workflows like software code for easier collaboration and tracking of changes.
File processing capabilities
Files processing including CSV and JSON.
Versioning
Version workflows, tasks and data, rollback and execute a previous version track changes over time.
Reusability
Define generic tasks and workflows that can be used in different contexts.
Data validation
Ensures incoming and outgoing data in the pipeline steps comply with schema and quality constraints.
UI testing
Execution testing directly in the interface for fast Q&A before deployment.
Debugging
Advanced debugging tools directly in the flow editing interface for faster identification and troubleshooting.
No more deployment complexity
No need to create a dev environment to package and deploy an application, just work on the browser and use CI/CD.
Accelerate Development and Deployment
Build Faster, Ship Better
CI/CD Deployment
Integrate Kestra into CI/CD pipelines to automate the deployment of workflows (Terraform, GitLab Actions, GitLab, Jenkins…).
Production made easy, no more production plan
Kestra supports flow dependencies to define the execution order based on their interdependence.
Real-Time Events
Flow execution based on events from the core or plugins, such as the result of other flows in real-time, detection of files from SFTP or Google Cloud Storage, or results of an SQL query.
Event-driven Triggers
Endless possibilities to start a flow through API, using a webhook, waiting for a trigger, or simply starting directly from the Web UI.
Monitoring
Dashboards and metrics directly in the User Interface.
Error Handling
Handle errors automatically and retry failed tasks.
Notifications
Alerting configuration directly in the UI to alert when workflow tasks are completed, have errors or delays, or for any other important event.
Large Data Processing
Massive data volume handling with high performance using partitioning and batch processing techniques.
Micro-Batch for Real-Time
All the benefits of a real-time system without maintenance and complexity costs by collecting data at regular intervals and processing them in micro-batch.
No single point of failure
Horizontal and vertical scaling without worrying about resource allocation.
Performance Gains
Better performance with JVM (Java Virtual Machine) technology.
Centralized Data Control for Improved Efficiency
Data at Heart with Internal Storage
Abstraction of Files
The internal Storage feature abstracts files, enabling each task to produce or consume files without needing to know the server's current location.
Stateless Servers
Allows easy destruction, restart from scratch or restart a previous execution without any hassle.
Metadata Storage
Metadata from the flow is stored on a database and Kestra's Internal Storage for transparent file systems.
Strongly Typed
Ensure each piece of data is the right type and well-defined, a date must be a date, an int must be an int, and so on…
Amazon Ion Integration
Handles more types than JSON, to enable easy mixing of tasks usage without any pain.
Communication Between Tasks
Each task can easily communicate with all others, making it simple to mix tasks usage without any problems.
Task Switch flexibility
All switch between tasks, such as BigQuery to MySQL or BigQuery to Redshift, without needing specific operators.
Security and high availability
Take Kestra to the next level with the Enterprise Edition 🚀
Service account
Create service accounts allowing to trigger workflows for external systems.
Secrets
Integrate your secret manager to ensure the security of your passwords and keys.
Audit Log
Audit all the resources created in one instance.