​Monitoring & ​Alerting

Kestra will deploy a monitoring endpoint on port 8081 by default. (You can change this port with configuration options endpoints.all.port)

This monitoring endpoint will expose some helpful routes in order to monitor Kestra:

Prometheus

If you have any Prometheus compatible monitoring system, Kestra exposes Prometheus metrics on the endpoint /prometheus.

Kestra metrics

Kestra exposes some internal metrics allowing to add some alerts. Each metric declares many timeseries with tags allowing to track at least namespace & flow. But also some other tags depending on available tasks.

Each task type can expose custom metrics that will be exposed on Prometheus.

Worker

MetricsTypeDescription
worker.running.countGAUGECount of tasks actually running
worker.started.countCOUNTERCount of tasks started
worker.retried.countCOUNTERCount of tasks retried
worker.ended.countCOUNTERCount of tasks ended
worker.ended.durationTIMERDuration of tasks ended

Executor

MetricsTypeDescription
executor.taskrun.next.countCOUNTERCount of tasks found
executor.taskrun.ended.countCOUNTERCount of tasks ended
executor.taskrun.ended.durationTIMERDuration of tasks ended
executor.workertaskresult.countCOUNTERCount of task result send by worker
executor.execution.started.countCOUNTERCount of executions started
executor.execution.end.countCOUNTERCount of executions ended
executor.execution.durationTIMERDuration of executions ended

Indexer

MetricsTypeDescription
indexer.countCOUNTERCount of indexation sent to repository
indexer.durationDURATIONDuration of indexation sent to repository

Scheduler

MetricsTypeDescription
scheduler.trigger.countCOUNTERCount of trigger found
scheduler.evaluate.running.countCOUNTEREvaluation of trigger actually running (aka: number of threads used by the scheduler)
scheduler.evaluate.durationTIMERDuration of evaluation of trigger

Others metrics

It will also expose all internal metrics from :

Some additional information can be found on Micronaut documentation

Grafana or Kibana

Since Elasticsearch is used to store all executions & metrics, you can easily make a dashboard with Grafana or Kibana in order to follow your Kestra instance.

In a near future, we will provide a template dashboard as a quick start.

Kestra endpoints

Kestra exposes some internal endpoints on the management port (8081 by default), depending on servers:

  • /worker: will expose all currently running tasks on this worker.
  • /scheduler: will expose all currently scheduled flows on this scheduler with the next date.
  • /kafkastreams: will expose all Kafka Streams states and aggregated store lag.
  • /kafkastreams/{clientId}/lag: will expose details lag for a clientId.
  • /kafkastreams/{clientId}/metrics: will expose details metrics for a clientId.

Other Micronaut default endpoints

Since Kestra is based on Micronaut, the default Micronaut endpoints are enabled by default on port 8081 :

  • /info Info Endpoint with git status information.
  • /health Health Endpoint usable as an external heathcheck for the application.
  • /loggers [Loggers Endpoint allows changing logger level at runtime.
  • /metrics Metrics Endpoint metrics in JSON format.
  • /env Env Endpoint to debug configuration files.

You can disable some endpoints following Micronaut configuration above.