background
Recently responsible for a self-developed Dubbo The registry often receives CPU Usage alarm , So a wave of optimization , It's not bad , So I'm going to share my thoughts 、 The optimization process , I hope it can help you .
Since the research Dubbo What is a registry , I'll draw a diagram. Just feel it a little , It doesn't matter if you don't understand , Does not affect subsequent understanding .
- Consumer and Provider Service discovery request （ register 、 Cancellation 、 subscribe ） All to Agent, It is its sole agent
- Registry and Agent keep Grpc Long link , The main purpose of long links is Provider In case of any change , It can be pushed to the corresponding Consumer. In order to ensure the correctness of the data , Made a push-pull combination mechanism ,Agent Will go to... Every once in a while Registry Pull the subscribed service list
- Agent And business services are deployed on the same machine , similar Service Mesh The idea of , Minimize intrusion into the business , So you can iterate quickly
Back to today's point , This registry recently CPU The utilization rate is in the middle and high water level for a long time , There are occasional app releases , When the push volume is large ,CPU It will even be full .
I haven't felt it before , Because there are not many access applications , In recent months, there have been more and more applications , The alarm threshold is slowly reached .
Looking for optimization points
Because this project is Go Written （ Don't understand, Go My friends don't matter , This paper focuses on the optimization of the algorithm , Not in the use of tools ）, Find out where CPU It's quite simple ： open pprof that will do , Go online to collect for a period of time .
For specific operation, please refer to my previous This article , The knowledge and tools used in today's article , This article can be found .
CPU profile Cut part of the picture , The others are less important , You can see the consumption CPU What's more AssembleCategoryProviders Method , Directly related to it is
- 2 individual redis Related methods
- 1 It's called
assembleUrlWeightMethods
Explain a little bit ,AssembleCategoryProviders The method is to construct a return Dubbo provider Of url, Because it will return url Do something about it （ For example, adjust the weight ）, It will involve understanding this Dubbo url Parsing . And because of the push-pull combination mode , The more users of online services , This one deals with QPS The greater the , So it takes up most of CPU No wonder .
these two items. redis The operation may be that serialization takes up CPU, The bigger head is assembleUrlWeight, It's a bit of a puzzle .
Next, let's analyze assembleUrlWeight How to optimize , Because he occupied CPU most , The optimization effect must be the best .
Here is assembleUrlWeight The pseudo code ：
func AssembleUrlWeight(rawurl string, lidcWeight int) string {
u, err := url.Parse(rawurl)
if err != nil {
return rawurl
}
values, err := url.ParseQuery(u.RawQuery)
if err != nil {
return rawurl
}
if values.Get("lidc_weight") != "" {
return rawurl
}
endpointWeight := 100
if values.Get("weight") != "" {
endpointWeight, err = strconv.Atoi(values.Get("weight"))
if err != nil {
endpointWeight = 100
}
}
values.Set("weight", strconv.Itoa(lidcWeight*endpointWeight))
u.RawQuery = values.Encode()
return u.String()
}
The ginseng rawurl yes Dubbo provider Of url,lidcWeight It's the weight of the computer room . According to the configured machine room weight , take url Medium weight Recalculate , Realize the distribution of multi machine room flow according to weight .
This process involves url Parameter parsing , Proceed again weight The calculation of , Finally, it is restored to a url
Dubbo Of url Structure and common url The structure is consistent , Its characteristic is that there may be many parameters , No, # The following fragment .
CPU It is mainly consumed in these two parsing and final restore , The purpose of these two analyses is to get url Medium lidc_weight and weight Parameters .
url.Parse and url.ParseQuery All are Go Official library , Each language has its own implementation , Its core is to analyze url For an object , To obtain conveniently url Every part of .
If you understand the concept of information entropy , In fact, you probably know that it must be optimized .Shannon（ Shannon ） Drawing on the concept of Thermodynamics , The average amount of information excluding redundancy is called Information entropy .
url.Parse and url.ParseQuery In this scenario, there must be redundancy in parsing , Redundancy means CPU Doing superfluous things .
Because a Dubbo url Parameters are usually many , We just need to take these two parameters , and url.Parse All parameters are parsed .
for instance , Given an array , Find the maximum of , If you sort the array first , Taking the maximum value again obviously has redundant operation .
The sorted array can not only take the maximum value , It can also take the second largest value 、 Third largest value ... minimum value , Information is redundant , So sorting first is definitely not the optimal solution for the maximum .
Optimize
Optimize access to url Parameter performance
The first idea is , Don't parse all url, Just take the corresponding parameters , This is very much like the algorithm problem we wrote , Like getting weight Parameters , It can only be these two cases （ non-existent #, So it's a lot simpler ）：
- dubbo://127.0.0.1:20880/org.newboo.basic.MyDemoService?weight=100&...
- dubbo://127.0.0.1:20880/org.newboo.basic.MyDemoService?xx=yy&weight=100&...
Or &weight=, Or ?weight=, The end is either &, Or go straight to the end of the string , The code is easy to write , First write an algorithm to analyze parameters ：
func GetUrlQueryParam(u string, key string) (string, error) {
sb := strings.Builder{}
sb.WriteString(key)
sb.WriteString("=")
index := strings.Index(u, sb.String())
if (index == -1) || (index+len(key)+1 > len(u)) {
return "", UrlParamNotExist
}
var value = strings.Builder{}
for i := index + len(key) + 1; i < len(u); i++ {
if i+1 > len(u) {
break
}
if u[i:i+1] == "&" {
break
}
value.WriteString(u[i : i+1])
}
return value.String(), nil
}
The original method of obtaining parameters can be picked out ：
func getParamByUrlParse(ur string, key string) string {
u, err := url.Parse(ur)
if err != nil {
return ""
}
values, err := url.ParseQuery(u.RawQuery)
if err != nil {
return ""
}
return values.Get(key)
}
Let's start with these two functions benchmark：
func BenchmarkGetQueryParam(b *testing.B) {
for i := 0; i < b.N; i++ {
getParamByUrlParse(u, "anyhost")
getParamByUrlParse(u, "version")
getParamByUrlParse(u, "not_exist")
}
}
func BenchmarkGetQueryParamNew(b *testing.B) {
for i := 0; i < b.N; i++ {
GetUrlQueryParam(u, "anyhost")
GetUrlQueryParam(u, "version")
GetUrlQueryParam(u, "not_exist")
}
}
Benchmark give the result as follows ：
BenchmarkGetQueryParam-4 103412 9708 ns/op
BenchmarkGetQueryParam-4 111794 9685 ns/op
BenchmarkGetQueryParam-4 115699 9818 ns/op
BenchmarkGetQueryParamNew-4 2961254 409 ns/op
BenchmarkGetQueryParamNew-4 2944274 406 ns/op
BenchmarkGetQueryParamNew-4 2895690 405 ns/op
You can see that the performance has improved 20 Many times
The newly written method , There are two small details , The first is to distinguish whether the parameter exists in the return value , This one will use ; The second is the string operation, which uses strings.Builder, This is also the result of the actual test , Use + perhaps fmt.Springf The performance is not as good as this , If you are interested, you can test it .
Optimize url Write parameter performance
To calculate the weight I'll put it later weight write in url in , The optimized code is directly given here ：
func AssembleUrlWeightNew(rawurl string, lidcWeight int) string {
if lidcWeight == 1 {
return rawurl
}
lidcWeightStr, err1 := GetUrlQueryParam(rawurl, "lidc_weight")
if err1 == nil && lidcWeightStr != "" {
return rawurl
}
var err error
endpointWeight := 100
weightStr, err2 := GetUrlQueryParam(rawurl, "weight")
if weightStr != "" {
endpointWeight, err = strconv.Atoi(weightStr)
if err != nil {
endpointWeight = 100
}
}
if err2 != nil { // url Does not exist in the weight
finUrl := strings.Builder{}
finUrl.WriteString(rawurl)
if strings.Contains(rawurl, "?") {
finUrl.WriteString("&weight=")
finUrl.WriteString(strconv.Itoa(lidcWeight * endpointWeight))
return finUrl.String()
} else {
finUrl.WriteString("?weight=")
finUrl.WriteString(strconv.Itoa(lidcWeight * endpointWeight))
return finUrl.String()
}
} else { // url in weight
oldWeightStr := strings.Builder{}
oldWeightStr.WriteString("weight=")
oldWeightStr.WriteString(weightStr)
newWeightStr := strings.Builder{}
newWeightStr.WriteString("weight=")
newWeightStr.WriteString(strconv.Itoa(lidcWeight * endpointWeight))
return strings.ReplaceAll(rawurl, oldWeightStr.String(), newWeightStr.String())
}
}
It is mainly divided into url Whether there is weight Two situations to discuss ：
- url There is no such thing as weight Parameters , Directly in url One at the back weight Parameters , Of course, pay attention to whether there is
? - url It exists in itself weight Parameters , String replacement is performed directly
Careful, you must have found , When lidcWeight = 1 when , Go straight back to , because lidcWeight = 1 when , The latter calculations actually don't work （Dubbo The default weight is 100）, Just don't operate , Save it CPU.
All optimized , Do it overall benchmark：
func BenchmarkAssembleUrlWeight(b *testing.B) {
for i := 0; i < b.N; i++ {
for _, ut := range []string{u, u1, u2, u3} {
AssembleUrlWeight(ut, 60)
}
}
}
func BenchmarkAssembleUrlWeightNew(b *testing.B) {
for i := 0; i < b.N; i++ {
for _, ut := range []string{u, u1, u2, u3} {
AssembleUrlWeightNew(ut, 60)
}
}
}
give the result as follows ：
BenchmarkAssembleUrlWeight-4 34275 33289 ns/op
BenchmarkAssembleUrlWeight-4 36646 32432 ns/op
BenchmarkAssembleUrlWeight-4 36702 32740 ns/op
BenchmarkAssembleUrlWeightNew-4 573684 1851 ns/op
BenchmarkAssembleUrlWeightNew-4 646952 1832 ns/op
BenchmarkAssembleUrlWeightNew-4 563392 1896 ns/op
Maybe it's going up 18 Multiple performance , And this may still be a bad situation , If you pass in lidcWeight = 1, better .
effect
Optimization finished , Write the corresponding unit test for the changed method , After confirming that there is no problem , Go online to observe ,CPU Idle（ Idle rate ） Promoted 10% above
Last
In fact, this article shows a Go Program very routine performance optimization , It's also relatively simple , After the play , You may have questions ：
- Why do I need to analyze when pushing and pulling url Well ？ Can't you calculate it in advance and save it ？
- Why only optimize this , Can other points be optimized ？
For the first question , In fact, this is a historical problem , That's how he was when you took over the system , If something goes wrong with the program , You change the whole mechanism , The cycle may be long , And it's easy to go wrong
The second question is , In fact, I just answered by the way , This optimizes , The smallest change , The most profitable , Other points are not so easy to change , In the short term , The most important thing is to get income . Of course, we also intend to reconstruct the system later , But before refactoring , This optimizes , Enough to solve the problem .
