! This post is also available in the following language. Korean

Monitoring Prometheus metrics from Armeria

In this post, we’ll be taking a look at how you can monitor Prometheus metrics collected with Armeria. For those of you who are trying Armeria for the first time, I will be adding simple practice samples so that you can follow along.

 

Practice exercise

The practice session was designed on macOS Big Sur. Let’s now take a look at how you should install and configure Gradle, Prometheus, and Grafana to run the samples, and see what the results are.

 

Configuring Gradle

I’ve defined the following dependencies on the ‘build.gradle’ file to collect the Prometheus metrics from the Armeria server.

  • build.gradle
dependencies {
 
    // Armeria
    implementation "com.linecorp.armeria:armeria:1.8.0"
    implementation "com.linecorp.armeria:armeria-logback:1.8.0"
 
    // Prometheus
    implementation "io.micrometer:micrometer-registry-prometheus:1.7.0"
 
    // ...
}

I’ve also prepared a simple REST API that responds to a /hello/{seq} GET request with “Success” or “Failure” for the purpose of our exercise.

  • MyAnnotatedService.java
public class MyAnnotatedService {
 
    @Get("/hello/{seq}")
    public HttpResponse hello(@Param("seq") int seq) {
        if (seq % 3 == 0) {
            return HttpResponse.of(HttpStatus.INTERNAL_SERVER_ERROR);
        }
        return HttpResponse.of(HttpStatus.OK, MediaType.PLAIN_TEXT_UTF_8, "Success");
    }
}

Below is the main() function code for running an Armeria server

  • ArmeriaPrometheusApplication.java
public class ArmeriaPrometheusApplication {
 
    public static void main(String[] args) {
        PrometheusMeterRegistry meterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
        Server server = Server
                .builder()
                .http(8083)
                .meterRegistry(meterRegistry)
                .annotatedService(new MyAnnotatedService())
                .service("/metrics", PrometheusExpositionService.of(meterRegistry.getPrometheusRegistry()))
                .decorator(MetricCollectingService.builder(MeterIdPrefixFunction.ofDefault("my.http.service"))
                                                  .newDecorator())
                .build();
 
        CompletableFuture<Void> future = server.start();
        future.join();
    }
}

You must always add MetricCollectingService as a decorator if you want to collect service metrics. (You can find more on how to do this in the official Armeria documentation.). You can customize the default prefix of the metrics by specifying the parameter of the MeterIdPrefixFunction.ofDefault() function.

  • ArmeriaPrometheusApplication.java
.decorator(new MyAnnotatedService(),
           MetricCollectingService.builder(MeterIdPrefixFunction.ofDefault("my.http.service"))
                                  .newDecorator())

Use PrometheusExpositionService to configure the path to output the Prometheus metrics.

  • ArmeriaPrometheusApplication.java
.service("/metrics", PrometheusExpositionService.of(meterRegistry.getPrometheusRegistry()))

Send a request with the /hello API to check if there is a proper response.

$ curl http://localhost:8083/hello/1
Success
 
$ curl http://localhost:8083/hello/3
500 Internal Server Error

Also check if a request with the /metrics API properly returns the Prometheus metrics as a response.

$ curl -s http://localhost:8083/metrics | grep "my_http_service_requests_total"
# HELP my_http_service_requests_total
# TYPE my_http_service_requests_total counter
my_http_service_requests_total{hostname_pattern="*",http_status="500",method="hello",result="success",service="com.example.armeria_prometheus.MyAnnotatedService",} 0.0
my_http_service_requests_total{hostname_pattern="*",http_status="200",method="hello",result="failure",service="com.example.armeria_prometheus.MyAnnotatedService",} 0.0
my_http_service_requests_total{hostname_pattern="*",http_status="200",method="hello",result="success",service="com.example.armeria_prometheus.MyAnnotatedService",} 1.0
my_http_service_requests_total{hostname_pattern="*",http_status="500",method="hello",result="failure",service="com.example.armeria_prometheus.MyAnnotatedService",} 1.0

 

Installing and configuring Prometheus

First install Prometheus with the Homebrew package manager.

$ brew install prometheus

 

When the installation process is finished, a file named ‘prometheus.yml’ is created. You can use this YAML file to change Prometheus configurations. 

$ vi $(brew --prefix)/etc/prometheus.yml

Carefully enter the paths and ports for collecting the Prometheus metrics in the configuration file.

  • prometheus.yml
global: 
  scrape_interval: 10s
 
scrape_configs:
  - job_name: "armeria-prometheus"
    metrics_path: '/metrics'
    static_configs:
    - targets: ["localhost:8083"]

Once you have made the changes, you must restart the Prometheus service to apply the changed settings.

$ brew services start prometheus

Prometheus uses the 9090 port by default. Navigate to http://localhost:9090 on your browser and you will see the screen shown below.

Enter “Armeria” into the search field to check if the related metrics have been properly collected from the Prometheus server.

Try sending multiple /hello/{seq} API requests and see if the my_http_service_requests_total metric properly increases.

 

Installing and configuring Grafana

Install and run the Grafana service using Homebrew the same way you installed Prometheus.

$ brew install grafana
$ brew services start grafana

Grafana uses the 3000 port by default. Navigate to http://localhost:3000 on your browser and you will see the login screen shown below.

Don’t worry if you have never created an account. You can enter ‘admin’ on both the Email or username and Password field to log in. As you can probably tell from the name, “admin” is a very basic account, and you will need to create a proper new account when you actually run the service in a real environment.

Enter the path to Prometheus in the Data Source field to display the Prometheus metrics as a graph on Grafana.

Next, create a dashboard and then add a panel where you can monitor the number of successful and unsuccessful attempts of the /hello/{seq} API.

The queries are as follows.

  • 200: increase(my_http_service_requests_total{hostname_pattern="*", method="hello", http_status="200", result="success", service="com.example.armeria_prometheus.MyAnnotatedService"}[1m])
  • 500: increase(my_http_service_requests_total{hostname_pattern="*", method="hello", http_status="500", result="failure", service="com.example.armeria_prometheus.MyAnnotatedService"}[1m])

The armeria_server_requests_total metric is a Counter-type metric that only continues to increase until you reset it. Because of this, the cumulative amount of times the API call succeeded or failed doesn’t provide much insight when monitoring in a real environment. What you should instead focus on, is how many times it either succeeded or failed in a given time frame . For this purpose, you can use the increase() function to express the number of successful and unsuccessful attempts per minute as a graph.

 

Test results

Let’s now run the client code below to send 100 API requests. (We used Armeria’s WebClient)

  • HttpClientApplication.java
public class HttpClientApplication {
 
    private static final Logger logger = LoggerFactory.getLogger(HttpClientApplication.class);
 
    public static void main(String[] args) {
        WebClient client = WebClient.of("http://localhost:8083/");
        for (int i = 0; i < 100; i++) {
            AggregatedHttpResponse res = client.get("/hello/" + i).aggregate().join();
            logger.info(res.content(StandardCharsets.UTF_8));
        }
    }
}

You can see the results of calling the /hello/{seq} API being expressed as a beautiful graph on Grafana.

 

Closing words

In this post, we took a look at how you can monitor Prometheus metrics collected from running an Armeria server, using a Grafana dashboard. Armeria provides various features for collecting metrics, and in our next post we will take a look at how you can customize the metrics you collect from Armeria to your needs.