Briefly describe the quantitative analysis system of services

Quantitative analysis system is to quantify the system based on data 、 Positioning and analysis , Then corresponding governance measures are produced , Guide the next online and offline governance , Mainly from the measurement 、 The quantitative analysis system is discussed from the dimensions of positioning and risk analysis .

1、 Stability risk measurement

The current stability of the service has always been , It's not easy to measure and judge . In order to grasp the real situation of stability construction , At the same time, guide and standardize the practices of business personnel in stability construction , For some important aspects of stability , It is necessary to formulate certain measurement standards , Measure the daily stability construction of business personnel , Clearly tell you the current level of stability work , What specific areas need to be improved .

For example, a quantitative standard can be formulated in combination with the change specification —— Change the credit score . According to the changed credit scoring standard , Make a systematic analysis of the weekly online orders , And summarize by team 、 Quantification and ranking , So that everyone can see the overall situation of the problem and their respective severity from the overall perspective , And it can index the specific change orders of each team from top to bottom 、 Change people and change parameters , Even until the specific change module configuration interface , So as to promote each team to find out the change risk and clearly know how to improve .

For example, monitoring alarm , Whether there is missing configuration for basic monitoring 、 Upstream and downstream depend on whether health is complete 、 Whether the alarm strategy meets the requirements , Can launch monitoring health score , Quantify the completeness of service monitoring alarms , Guide users to improve the monitoring and alarm . For example, plan construction , You can launch the health score of the plan , Quantify the construction of a service plan , Whether the plans involving degradation and current limitation are complete , Whether the implementation of the plan meets the gray requirements , Whether the plan can be rolled back .

For stability measures , It is mainly to determine specific measurement indicators , And the standard score of each measurement . In fact, the specific score is not so important , The key is that everyone can agree on the measurement standard , Be able to have a practical quantitative guidance effect on the daily work of stability .

2、 Fault location based on multi-dimensional monitoring

When there is a problem with online services , When there is enough monitoring information , To intuitively locate the problem . But as the business scale grows , The number of microservices is increasing , link 、 Topology 、 The network is becoming more and more complex , There are more and more corresponding monitoring events . When a fault occurs , A large number of alarm messages may appear instantly , Quickly find the cause of the fault from many alarms , Determine the corresponding stop loss plan , Is a very important and challenging thing .

Failure time , First of all, we need to determine the influence surface of the fault , The business can be organized into a global one based on scenarios and distributed tracking topology “ Fire fighting diagram ”, The fire fighting diagram includes the availability indicators of all core services , For example, error rate 、QPS、 Time consuming, etc , Failure time , First, see the business and service affected by the fault from the fire-fighting diagram , Next, determine the scope of fault location .

In order to locate the specific reason from the numerous and complex events , You can alarm the monitoring of each dimension 、 Various change events take the form of events , Integrate into a timeline according to the timeline , With the event timeline , We can focus on the monitoring alarm events and change events in the period before the failure time , So as to determine the corresponding plan and stop loss measures according to the specific event type .

3、 risk analysis

Based on online real-time observable data , And the change and operation data of the whole R & D process , We can get many dimensional reports and trend data . These dimensions can cover all links of service governance , For example, link SLA、 Time out to try again 、 Capacity management 、 Strength dependence, etc , These data can be used as the basis for the next analysis .

meanwhile , According to the previous risk analysis and some static service metadata information , It will form a historical benchmark library corresponding to the current real-time governance data , Compare the current data with the historical benchmark Library , Find trends and laws , And then discover the potential risks .

After real-time governance data analysis, some valuable things can be deposited into the historical benchmark Library , As the basis for subsequent risk analysis , And then form a closed-loop mechanism of risk analysis . In online and offline governance based on risk analysis , Will conduct a comprehensive analysis in combination with specific scenarios and examples .