Grpc: how to add API Prometheus monitoring interceptors / Middleware?

Introduce

This article will show you how to gRPC Add in microservice API Prometheus( Prometheus ) Interceptor / middleware . That is to say, it can be in Grafana Made in API monitor .

What is? API Prometheus( Prometheus ) Interceptor / middleware ？

Prometheus( Prometheus ) The interceptor will API Request record Prometheus( Prometheus ) monitor .

We will use rk-boot To start up gRPC service .

Please visit the following address for a complete tutorial ：https://rkdev.info/cnhttps://rkdocs.netlify.app/cn ( spare )

install

go get github.com/rookie-ninja/rk-boot
go get github.com/rookie-ninja/rk-grpc

Quick start

rk-boot By default, the following two open source libraries are integrated .

• rk-prom As Prometheus (Prometheus) Client startup Library .

Be careful ！ In order for the example to proceed smoothly , Please be sure to in go.mod In the document ,module The suffix of is set to rk-demo. for example ：module github.com/your-repo/rk-demo

1. establish boot.yaml

In order to verify , We launched the following options ：

• commonService：commonService It contains a series of general API. details
• prom：Prometheus( Prometheus ) client .
• grpc-gateway：grpc It starts by default grpc-gateway To provide Restful API service . At the time of verification , We can send... Directly Restful request .
• prometheus middleware ： start-up prometheus middleware .

---
grpc:
  - name: greeter                   # Name of grpc entry
    port: 8080                      # Port of grpc entry
    enabled: true                   # Enable grpc entry
    commonService:
      enabled: true                 # Enable common service for testing
    prom:
      enabled: true                 # Enable prometheus client
    interceptors:
      metricsProm:
        enabled: true               # Enable prometheus interceptor

2. establish main.go

package main

import (
	"context"
	"github.com/rookie-ninja/rk-boot"
	_ "github.com/rookie-ninja/rk-grpc/boot"
)

// Application entrance.
func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Bootstrap
	boot.Bootstrap(context.Background())

	// Wait for shutdown sig
	boot.WaitForShutdownSig(context.Background())
}

3. Folder structure

$ tree
.
├── boot.yaml
├── go.mod
├── go.sum
└── main.go

0 directories, 4 files

4. verification

$ go run main.go

Send a request

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

visit Prometheus client : http://localhost:8080/metrics

Visual monitoring

We have started in the local process prometheus monitor , The rest is how to be in a 【 beautiful 】 Check the monitoring on our website .

There are many tools on the market , But we choose 【 Simple 】,【 popular 】,【 free 】 The way , That is to say Prometheus + Grafana.

Architecture diagram

Let's take a look at the whole process .

In fact, the principle is very simple , Namely 【 hijacked 】API request , And record 【 Time 】,【 Error code 】 Etc . after , Give Way Prometheus The service takes the initiative from 【 Services created 】 in , Pull data . Last , Give Way Grafana Service from Prometheus Middle pull data , Show data sheet .

Quick start

1. establish prometheus.yml

Let's create prometheus.yml The configuration file , Give Way prometheus Services from localhost:8080/metrics Pull data .

In the following configuration , We didn't specify /metrics, because prometheus By default /metrics To pull data .

Be careful ！ We put targets Set up a host.docker.internal:8080 instead of localhost:8080, This is because prometheus In the container , Our service is local .

Want to access the port of the local machine from the container , This is a convenient way . explain

global:
  scrape_interval: 1s # Make scrape interval to 1s for testing.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'rk-demo'
    scrape_interval: 1s
    static_configs:
      - targets: ['host.docker.internal:8080']

2. start-up Prometheus

We use docker To start up .

Prometheus By default 9090 port .

$ docker run \
      -p 9090:9090 \
      -v /<your path>/rk-demo/prometheus.yml:/etc/prometheus/prometheus.yml \
      prom/prometheus

3. verification Prometheus

Please follow the above 【 verification 】, start-up main.go, And send a /rk/v1/healthy request .

then , Let's take a look prometheus Data in the service .

visit ：localhost:9090, And search rk_demo_greeter_resCode, We can see that the data is already prometheus Inside the .

visit ：localhost:9090/targets, We can see ,prometheus Data has been pulled every second .

4. start-up Grafana

Grafana By default 3000 port .

$ docker run -p 3000:3000 --name grafana grafana/grafana

visit ：localhost:3000

Now ,grafana Will let you log in , The default user name and password are as follows .

user name ：admin password ：admin

5. stay Grafana Add in Prometheus data source

Grafana Just one. Web UI Tools , In order to see the data report , We tell Grafana Where to find Prometheus.

choice Prometheus As a data source .

Fill in Prometheus Address , The same reason as above , because Grafana Running on the Docker in , therefore , We don't use localhost:9090, It is ,host.docker.internal:9090.

6. Import Dashboard

We can edit it ourselves Grafana Dashboard, however , This is not an easy task . For use rk-boot Services started , We provide default 【 free 】 Of Grafana Dashboard Template .

Be careful , The imported Dashboard Only match 【 The service created according to the above logic 】.

Why? because rk-boot By default <App name >_<Entry name >_xxx As prometheus Of metrics name .

If the user uses a different module, Or different Entry name , It needs to be changed Dashboard Inside Variable. We'll see that in a later article , How to use Grafana.

Move to Dashboard Import page

Import 15111 Number Dashboard, The definition is ：https://grafana.com/grafana/dashboards/15111

Appoint Prometheus data source , This data source is configured above Prometheus.

Start monitoring

Be careful ！ If the number of requests is too small , It will be displayed as 0, Please send a few more requests .

Concept

We can start with Grafana Got surveillance data in the , Now look at rk-boot Middleware in , What type of monitoring data has been added .

The monitor interceptor will record the following by default .

Monitoring item	data type	details
elapsedNano	Summary	RPC Time consuming
resCode	Counter	be based on RPC Counter of return code
errors	Counter	be based on RPC Wrong counter

The above three monitoring , They all have the following labels .

label	details
entryName	gRPC entry name
entryType	gRPC entry type
realm	environment variable : REALM, eg: rk
region	environment variable : REGION, eg: beijing
az	environment variable : AZ, eg: beijing-1
domain	environment variable : DOMAIN, eg: prod
instance	Local Hostname
appVersion	from AppInfoEntry obtain
appName	from AppInfoEntry obtain
restMethod	If started grpc-gateway, And the request is in http Sent by form , Will be recorded . eg: GET
restPath	If started grpc-gateway, And the request is in http Sent by form , Will be recorded . eg: /rk/v1/healthy
grpcService	GRPC The service name .eg: rk.api.v1.RkCommonService
grpcMethod	GRPC Method name .eg: Healthy
grpcType	GRPC type .eg: UnaryServer
resCode	Return code , eg: OK