Monitoring & Alerting
Kestra will deploy a monitoring endpoint on port 8081 by default. (You can change this port with
configuration options endpoints.all.port)
This monitoring endpoint will expose some helpful routes in order to monitor Kestra:
Prometheus
If you have any Prometheus compatible monitoring system, Kestra exposes Prometheus metrics on the endpoint /prometheus.
Kestra metrics
Kestra exposes some internal metrics allowing to add some alerts. Each metric declares many timeseries with tags allowing to track at least namespace & flow. But also some other tags depending on available tasks.
Each task type can expose custom metrics that will be exposed on Prometheus.
Worker
| Metrics | Type | Description |
|---|---|---|
| worker.running.count | GAUGE | Count of tasks actually running |
| worker.started.count | COUNTER | Count of tasks started |
| worker.retried.count | COUNTER | Count of tasks retried |
| worker.ended.count | COUNTER | Count of tasks ended |
| worker.ended.duration | TIMER | Duration of tasks ended |
Executor
| Metrics | Type | Description |
|---|---|---|
| executor.taskrun.next.count | COUNTER | Count of tasks found |
| executor.taskrun.ended.count | COUNTER | Count of tasks ended |
| executor.taskrun.ended.duration | TIMER | Duration of tasks ended |
| executor.workertaskresult.count | COUNTER | Count of task result send by worker |
| executor.execution.started.count | COUNTER | Count of executions started |
| executor.execution.end.count | COUNTER | Count of executions ended |
| executor.execution.duration | TIMER | Duration of executions ended |
Indexer
| Metrics | Type | Description |
|---|---|---|
| indexer.count | COUNTER | Count of indexation sent to repository |
| indexer.duration | DURATION | Duration of indexation sent to repository |
Scheduler
| Metrics | Type | Description |
|---|---|---|
| scheduler.trigger.count | COUNTER | Count of trigger found |
| scheduler.evaluate.running.count | COUNTER | Evaluation of trigger actually running (aka: number of threads used by the scheduler) |
| scheduler.evaluate.duration | TIMER | Duration of evaluation of trigger |
Others metrics
It will also expose all internal metrics from :
Some additional information can be found on Micronaut documentation
Grafana or Kibana
Since Elasticsearch is used to store all executions & metrics, you can easily make a dashboard with Grafana or Kibana in order to follow your Kestra instance.
In a near future, we will provide a template dashboard as a quick start.
Kestra endpoints
Kestra exposes some internal endpoints on the management port (8081 by default), depending on servers:
/worker: will expose all currently running tasks on this worker./scheduler: will expose all currently scheduled flows on this scheduler with the next date./kafkastreams: will expose all Kafka Streams states and aggregated store lag./kafkastreams/{clientId}/lag: will expose details lag for a clientId./kafkastreams/{clientId}/metrics: will expose details metrics for a clientId.
Other Micronaut default endpoints
Since Kestra is based on Micronaut, the default Micronaut endpoints are enabled by default on port 8081 :
/infoInfo Endpoint with git status information./healthHealth Endpoint usable as an external heathcheck for the application./loggers[Loggers Endpoint allows changing logger level at runtime./metricsMetrics Endpoint metrics in JSON format./envEnv Endpoint to debug configuration files.
You can disable some endpoints following Micronaut configuration above.