Configure Monitoring with Grafana & Prometheus
Setup Prometheus and Grafana for monitoring Kestra.
We will look in detail at setting up monitoring dashboards for Kestra.
Kestra exposes Prometheus metrics at port 8081 on the endpoint /prometheus
. This endpoint can be used by any compatible monitoring system.
Use the docker-compose.yml file and start Kestra using the command:
% docker-compose up
Once Kestra is up and running, you can check out the available metrics by going to http://localhost:8081/prometheus
on your browser. The metrics should appear as below:
Create a few flows and execute them to generate some metrics for visualization. You can also add triggers to the flows to check the metrics corresponding to executions happening on a regular basis.
Setting up Prometheus
Now that the metrics are available from Kestra, we need to setup Prometheus server to scrape these metrics, and store them in a timeseries DB.
Firstly, let us create a prometheus.yml
file for scraping the metrics. Create a file prometheus.yml
with the following contents:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
metrics_path: /prometheus
static_configs:
- targets: ["<kestra-host-ip-address>:8081"]
Be sure to put the appropriate <kestra-host-ip-address>
in the last line. Now, we will start the Prometheus server using the following docker command:
% docker run -d -p 9090:9090 -v ~/Documents/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
You can now go to http://localhost:9090/graph
and try out visualizing some metrics using the PromQL. Here is one of the graphs for kestra_executor_execution_started_count_total
metric:
Setting up Grafana
Let us now move on to setting up Grafana. You start by installing Grafana using docker via the following command:
% docker run -d -p 3000:3000 --name=grafana grafana/grafana-enterprise
You can open the Grafana server at http://localhost:3000
. The default credentials are admin
as both username and passowrd. Once logged into Grafana, click on the hamburger menu on the top left and go to Connections -> Data Sources.
Click on Add new Data Source
button present on the top right, and select Prometheus
from the time series databases list. In the Prometheus server URL
text box, put in the following URL: http://<host-ip-address>:9090
. All the other confoguration can be left as default. You can click on Save and Test
button at the bottom, and confirm that the connection to Prometheus database is successful.
We are now all set to create the Grafana dashboard. For this, click on the +
button on the top of the page to add a New Dashboard
to Grafana. Save the dashboard with an appropriate name. Then, click on Add visualization
, and select prometheus
as the data source.
We will create a Gauge that shows the number of tasks that are presently running. For this, select Guage
as the Visualization in the top right corner. In the PromQL metrics explorer text box, you can write sum(kestra_worker_running_count)
. Click on Run queries
button to ensure the Guage shows up the number. You can now run some long-running task, and check that the Guage correctly reflects the count. You can now put an appropriate title in the Panel options, say Tasks running
. This is how your Grafana should look like:
Click on Save
and Apply
to add this guage to the dashboard.
Similarly, you can now keep on adding more graphs to your dashboard. Here is one of the sample dashboards for Kestra metrics.
Was this page helpful?