Take you hand in hand to develop a service monitoring component

Preface

With the rapid expansion of the business , More and more back-end teams adopt microservice design solutions . Microservice design reduces the threshold of business development , Yes ( Including surveillance ) Higher requirements for system infrastructure . In the micro service scenario , The number of back office services is expanding rapidly , The selection of various service technologies is diversified , There are many people involved , It brings great challenge to system monitoring .

This article will introduce a lightweight pluggable microservice monitoring plug-in from design to implementation .

Monitoring indicators

• CPU Usage rate
• JVM Memory usage
• Dubbo Number of thread pools
• Database connection pool usage

Principle of monitoring system

The principle of the monitoring system is generally divided into the following steps ：

• Data collection
• The data transfer 、 Storage
• The data processing 、 analysis 、 Handle
• Data presentation

This plug-in uses timed tasks , Collect data of various indicators , Analyze （ Compare with threshold ）, If the threshold is exceeded , The system will send warning information to the maintainer .

Next, the implementation of each module is introduced one by one .

CPU monitor

In limine , I am right. CPU The way of monitoring is to use Hyperic-Sigar.

It is a tool set for collecting the underlying information of the system , It is characterized by comprehensive information collection , It can collect CPU、MEM、NETWORK、PROCESS、IOSTAT And other resources . It also supports cross platform .

The important thing is that it encapsulates all kinds of resource information comprehensively , It is very convenient to use .

Configure before using . Different files can be selected according to the system and the system version .

pom rely on ：

<dependency>
    <groupId>org.hyperic</groupId>
    <artifactId>sigar</artifactId>
    <version>1.6.5.132</version>
</dependency>

Configuration class ：

@Slf4j
@Configuration
public class SigarConfig {
    
​
    static {
    
        initSigar();
    }
​
    /** *  initialization sigar Configuration file for  */
    public static void initSigar() {
    
        log.info("==initSigar==");
        SigarLoader loader = new SigarLoader(Sigar.class);
        String lib = null;
​
        try {
    
            lib = loader.getLibraryName();
            log.info("sigar lib:{}", lib);
        } catch (ArchNotSupportedException e) {
    
            log.error("error:", e);
        }
        ResourceLoader resourceLoader = new DefaultResourceLoader();
        Resource resource = resourceLoader.getResource("classpath:/sigar.so/" + lib);
        if (resource.exists()) {
    
            log.info("==exists==");
            InputStream is = null;
            BufferedOutputStream os = null;
            try {
    
                is = resource.getInputStream();
                File tempDir = new File("./log");
                if (!tempDir.exists()) {
    
                    tempDir.mkdirs();
                }
                os = new BufferedOutputStream(new FileOutputStream(new File(tempDir, lib), false));
                int lentgh = 0;
                while ((lentgh = is.read()) != -1) {
    
                    os.write(lentgh);
                }
​
                System.setProperty("org.hyperic.sigar.path", tempDir.getCanonicalPath());
            } catch (IOException e) {
    
                log.error("init siagr fail:", e);
            } finally {
    
                try {
    
                    if (is != null) {
    
                        is.close();
                    }
                    if (os != null) {
    
                        os.close();
                    }
                } catch (IOException e) {
    
                    log.error(" Closing error :", e.getMessage());
                }
            }
        }
    }
}

    public static CpuPerc[] cpu() {
    
        Sigar sigar = new Sigar();
        CpuPerc[] cpuList = null;
        try {
    
            cpuList = sigar.getCpuPercList();
        } catch (SigarException e) {
    
            log.error("error:", e);
        }
        return cpuList;
    }

Later, the monitoring period , It is found that this method has little effect .

Because our projects are deployed in containers , and sigar What you get is that of the host cpu usage . Such data is of no practical significance to us .

What we actually want to monitor is the container cpu usage .

Our statistics docker The resource usage of each container can be used docker stats command ：

Then how to use the program to realize ？——docker-java.

docker-java

Docker Of Java edition API.

Want to use Java Or other ways to access dockerAPI You need to set a port .

Run the following command , Get into docker.service：

vi /lib/systemd/system/docker.service

• find Execstart=/usr/bin/dockerd Place of action ;
• It is amended as follows tcp://0.0.0.0:2375 -H unix://var/run/docker.sock;
• Save and exit .

Then run the following command ：

systemctl daemon-reload
 
service docker restart// Restart docker
 
systemctl stats docker// You can view relevant contents , have a look 2375 Whether it has been set up 

You can pass java To visit docker 了 , Or you can access it in the address bar of the browser docker 了 .

Access... In the browser address bar ：http://ip:2375/info, The data returned is in json Format display of .

Use java When accessing, you need to introduce dependencies into the project ：

        <dependency>
            <groupId>com.github.docker-java</groupId>
            <artifactId>docker-java</artifactId>
            <version>3.1.3</version>
        </dependency>

DockerClientUtil：

    // Connect docker
    public DockerClient connectDocker() {
    
        DockerClient dockerClient = DockerClientBuilder.getInstance("tcp://localhost:2375").build();
        Info info = dockerClient.infoCmd().exec();
​
        log.info("docker The environmental information of is as follows :{}", JSONObject.toJSONString(info));
        return dockerClient;
    }

We passed a main Method to obtain the details of the resources occupied by the corresponding image ：

 public static void main(String[] args) {
    
​
        DockerClientUtil dockerClientService = new DockerClientUtil();
​
        // Connect docker The server 
        DockerClient client = dockerClientService.connectDocker();
        ContainerStatisticsInfo statistics = ContainerStatisticsInfo.builder().build();
        ResultCallback<Statistics> resultCallback = new ResultCallback<Statistics>() {
    
​
            @Override
            public void close() throws IOException {
    
            }
​
            @Override
            public void onStart(Closeable closeable) {
    
            }
​
            @Override
            public void onNext(Statistics object) {
    
                statistics.setStatistics(object);
​
            }
​
            @Override
            public void onError(Throwable throwable) {
    
            }
​
            @Override
            public void onComplete() {
    
            }
        };
        client.statsCmd("CONTAINER_ID").exec(resultCallback);
​
        try {
    
            Thread.sleep(2000);
        } catch (InterruptedException e) {
    
            e.printStackTrace();
        }
        System.out.println(JSONObject.toJSONString(statistics));
    }

Response information ：

{"ok":false,"statistics":{"blkioStats":{"ioMergedRecursive":[],"ioQueueRecursive":[],"ioServiceBytesRecursive":[{"major":8,"minor":0,"op":"Read","value":83476480},{"major":8,"minor":0,"op":"Write","value":13570048},{"major":8,"minor":0,"op":"Sync","value":13565952},{"major":8,"minor":0,"op":"Async","value":83480576},{"major":8,"minor":0,"op":"Total","value":97046528},{"major":253,"minor":0,"op":"Read","value":83476480},{"major":253,"minor":0,"op":"Write","value":13704704},{"major":253,"minor":0,"op":"Sync","value":13700608},{"major":253,"minor":0,"op":"Async","value":83480576},{"major":253,"minor":0,"op":"Total","value":97181184}],"ioServiceTimeRecursive":[],"ioServicedRecursive":[{"major":8,"minor":0,"op":"Read","value":988},{"major":8,"minor":0,"op":"Write","value":128},{"major":8,"minor":0,"op":"Sync","value":127},{"major":8,"minor":0,"op":"Async","value":989},{"major":8,"minor":0,"op":"Total","value":1116},{"major":253,"minor":0,"op":"Read","value":988},{"major":253,"minor":0,"op":"Write","value":128},{"major":253,"minor":0,"op":"Sync","value":127},{"major":253,"minor":0,"op":"Async","value":989},{"major":253,"minor":0,"op":"Total","value":1116}],"ioTimeRecursive":[],"ioWaitTimeRecursive":[],"sectorsRecursive":[]},"cpuStats":{"cpuUsage":{"percpuUsage":[1740906492,1576282396,2071067412,1788793435],"totalUsage":7177049735,"usageInKernelmode":3990000000,"usageInUsermode":1850000000},"onlineCpus":4,"systemCpuUsage":10680530000000,"throttlingData":{"periods":0,"throttledPeriods":0,"throttledTime":0}},"memoryStats":{"limit":3902095360,"maxUsage":474267648,"stats":{"activeAnon":376696832,"activeFile":19701760,"cache":95473664,"dirty":0,"hierarchicalMemoryLimit":9223372036854771712,"hierarchicalMemswLimit":9223372036854771712,"inactiveAnon":0,"inactiveFile":75771904,"mappedFile":21716992,"pgfault":93119,"pgmajfault":390,"pgpgin":78547,"pgpgout":51163,"rss":376696832,"rssHuge":358612992,"totalActiveAnon":376696832,"totalActiveFile":19701760,"totalCache":95473664,"totalDirty":0,"totalInactiveAnon":0,"totalInactiveFile":75771904,"totalMappedFile":21716992,"totalPgfault":0,"totalPgmajfault":0,"totalPgpgin":0,"totalPgpgout":0,"totalRss":376696832,"totalRssHuge":358612992,"totalUnevictable":0,"totalWriteback":0,"unevictable":0,"writeback":0},"usage":472170496},"networks":{"eth0":{"rxBytes":698,"rxDropped":0,"rxErrors":0,"rxPackets":9,"txBytes":0,"txDropped":0,"txErrors":0,"txPackets":0}},"pidsStats":{"current":37},"preCpuStats":{"cpuUsage":{"percpuUsage":[1738966229,1574918654,2069521792,1787287623],"totalUsage":7170694298,"usageInKernelmode":3990000000,"usageInUsermode":1850000000},"onlineCpus":4,"systemCpuUsage":10676500000000,"throttlingData":{"periods":0,"throttledPeriods":0,"throttledTime":0}},"read":"2021-08-10T14:39:39.345562286Z"}}

JVM Memory monitoring

JDK Provided java.lang.Runtime Class contains ：freeMemory()、totalMemory()、maxMemory(), Through these three methods , We can calculate JVM Proportion of memory used .

    public static JvmMemoryVO jvmMemory() {
    
        JvmMemoryVO jvmMemoryVO = new JvmMemoryVO();
        double totalMemory = Runtime.getRuntime().totalMemory() / 1024L / 1024L;
        double freeMemory = Runtime.getRuntime().freeMemory() / 1024L / 1024L;
        double maxMemory = Runtime.getRuntime().maxMemory() / 1024L / 1024L;
        // Java  The total amount of memory in the virtual machine   Company MB
        jvmMemoryVO.setTotalMemory(totalMemory);
        
        //Java  The amount of memory available in the virtual machine   Company MB
        jvmMemoryVO.setFreeMemory(freeMemory);
        
        // Java  The amount of memory used in the virtual machine   Company MB
        jvmMemoryVO.setUsedMemory(totalMemory - freeMemory);
        jvmMemoryVO.setMaxMemory(maxMemory);
        
        // Usage rate 
        jvmMemoryVO.setMemoryUsageRate(jvmMemoryVO.getUsedMemory()/ jvmMemoryVO.getTotalMemory());
        return jvmMemoryVO;
    }

Dubbo Thread pool monitoring

Dubbo It provides us with a Checker class ：ThreadPoolStatusChecker, Through it you can get Dubbo Thread pool information for .

I use reflection , call check Method , Then parse the returned data . Realize to Dubbo Thread pool monitoring .

    public static void checkDubbo() {
    
        //dubbo Thread pool number monitoring 
        Class<?> clazz = null;
        try {
    
            clazz = Class.forName("com.alibaba.dubbo.rpc.protocol.dubbo.status.ThreadPoolStatusChecker");
        } catch (ClassNotFoundException e) {
    
            log.error(" Class does not exist :", e);
        }
        Method check = null;
        try {
    
            check = clazz.getMethod("check");
        } catch (NoSuchMethodException e) {
    
            log.error(" Method does not exist :", e);
        }
        Object result = null;
        try {
    
            result = check.invoke(clazz.newInstance());
        } catch (IllegalAccessException e) {
    
            log.error("error:", e);
        } catch (InvocationTargetException e) {
    
            log.error("error:", e);
        } catch (InstantiationException e) {
    
            log.error("error:", e);
        }
    }

Database connection pool monitoring

The connection pool used in our project is Alibaba's DruidDataSource. The functions of its underlying package are very rich , This includes querying the relevant information of the database connection pool .

After the project starts , Type in the browser ：http://127.0.0.1:8080/druid/datasource.html Can access .

In the plug-in, I use Http call http://127.0.0.1:8080/druid/datasource.json To get the connection pool information .

It should be noted that , If the project is right druid Add Authentication ：

spring.datasource.druid.stat-view-servlet.login-username=admin
spring.datasource.druid.stat-view-servlet.login-password=admin

You need to request the login interface before accessing , Get cookie after , With this cookie To request .

Okay , That's all for this component , Students who need source code can confide in me .

At present, there are many mature monitoring components on the market ：spring-boot-actuator、elk etc. . Interested students can learn about it by themselves .

summary

• Develop basic components , First of all, ensure that the operation of the service is not affected . For exceptions, it is best to try/catch.
• Don't bring performance problems because of the introduction of components .
• The follow-up needs to be improved ： data storage 、kakfa Collect logs 、 Portrait analysis .