Monitor your app - Prometheus and Grafana

Written by Mohamed Elzomor, Software Engineer II in the Couch Potatoes squad at Kites Software

Monitor your app

Intro

The British physicist and mathematician, William Thomson Kelvin, among his statements, left us a phrase as revealing as the following: “What is not defined cannot be measured. What is not measured, cannot be improved. What is not improved, is always degraded”. We -as software developers- are always seeking to upgrade our projects whether in design, clean code or design patterns, etc.. But how can we measure the changes we have made? If you get into a messy legacy code, and you want to refactor that, how would you say that the refactored code is better, or at least gives the same performance? Perfect, the answer is by measuring.

Prometheus

Prometheus is an open-source monitoring and alarming toolkit, which was first developed by Soundcloud. The name refers to the titan known for his foresight and giving the fire to humanity in the Greek mythology, in that we see a symbol of enlightenment and knowledge, which aligns with the purpose of the application that brings visibility and insight into complex systems.

Prometheus collects and stores its metrics (Numerical measurements)as time series data (Changes over time), i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Setup Prometheus

The Prometheus server actively scrapes metrics from the application at intervals (This interval is customizable) so from the application side you only have to do two things:

1. Setup the Prometheus tool (In our case we use maven)
2. Expose the metrics API for the Prometheus server to read (Actuator API)

Exporters

But there might be some cases where you do not have access to the source code and you still want to monitor the application(For example, you want to monitor your Linux machine, or your PostgreSQL database). Prometheus introduced the (Exporter) which helps in generating the metrics in a form that Prometheus server can read and deal with (Node Exporter for Linux as an example).

Grafana

For now, this is perfect. However, I still can not see the results scraped by Prometheus, that’s when Grafana comes into play. Grafana is a powerful open-source project that is used mostly for visualizing metrics and logs. Also, to generate dashboards where you can easily monitor your application. It can be connected to different and multiple data sources (Like Prometheus, Graphite, Elastic Search) and generate dashboards using their metrics.

One of the most interesting parts in the dashboards, is that there is a variety of templates already existing on their official website, and all you have to do is only copying the ID of the template and pasting it in your application. Also, for a better customization, you can download the JSON file of the dashboard, customize it and use it in your own template.

Code time

1. Create docker-compose file for Prometheus and Grafana

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: always

 grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: 'admin'
    depends_on:
      - prometheus
    restart: always

2. In the same directory, create a new file name as in docker-compose file (prometheus.yml)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'spring_boot'
    metrics_path: "/actuator/prometheus"
    scrape_interval: 30s
    static_configs:
      - targets: ['host.docker.internal:8080']

3. Now both servers (localhost:9090) and (localhost:3000) shall be up and running with default credentials for Grafana is “admin” for username and “admin” for password that you have set in the docker-compose file

4. From the application side (Java, Spring Boot, Maven, .properties file)

4.a. Add maven dependency for Prometheus and Spring actuators (That exposes metrics) in the pom.xml file

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

4.b. Add in the application.properties file the following

management.endpoints.web.exposure.include=prometheus
management.endpoint.prometheus.path=/actuator/prometheus
management.endpoint.prometheus.enabled=true

5. Make sure the application and Prometheus are both up, then navigate to Prometheus (http://localhost:9090) to make sure the application is connected and up

6. Navigate to Grafana to add the Prometheus as a data source as following

6.a. Go to (http://localhost:3000) and login with (admin) (admin) you will be prompted to update your password

6.b. On the left pane, hit the (Data source) and add new data source (Prometheus)

6.c. Fill the data as following:

In the Prometheus Server URL ==> http://prometheus:9090

7. Now it is the time to create your dashboard, you can find lots of valuable templates on the Grafana website. Just choose the suitable template by copying the ID and putting it in your Grafana app (Or you can download the JSON for more customization)

8. Now you can monitor your application on Grafana through Prometheus.

Example for using Spring Boot statistics Template (ID: 15425)

Spring Boot Statistics template example

Conclusion

Monitoring has become a very essential part in the software development process, as you can never measure the effect of any upgrades you are making to the system without it. Prometheus and Grafana are a very customizable tools to help you doing so.