We were tossed all night by a Kong performance bug

The story background

stay Erda In the technical architecture of , We used kong As API Technology selection of gateway . Because it has the characteristics of high concurrency and low latency , Combined with Kubernetes Ingress Controller, Declarative configuration based on cloud native , Can achieve rich API Strategy .

In our earliest delivered cluster ,kong It is still relatively early 0.14 edition , With the increasing requirements of business level for security , We need to be based on kong Implement security plug-ins , Help the system to have better security capabilities . Due to the earlier 0.14 Version cannot be used go-pluginserver To expand kong Plug in mechanism of , We have to put kong Upgrade to a relatively new 2.2.0 edition .

The upgrade process will not be repeated here , Basically, it is upgraded smoothly step by step according to the official documents , But in the days after the upgrade , our SRE The team received intensive consultation and even criticism , Businesses deployed on the cluster Intermittently inaccessible , Very high latency .

A series of failed attempts

Parameter tuning

At first, in order to quickly fix this problem , We are right. kong Of NGINX_WORKER_PROCESSES、MEM_CACHE_SIZE、 DB_UPDATE_FREQUENCY、WORKER_STATE_UPDATE_FREQUENCY Parameters and postgres Of work_mem、 share_buffers Have been properly tuned .

however , No effect  .

Clean up the data

Due to the historical reasons of this cluster , Will register or delete frequently api data , Therefore, about 5 More than 10000 articles route perhaps service data .

We suspect that the performance degradation is caused by the large amount of data , And then combine erda Data pairs in kong Delete the historical data in , In the process of deletion, the deletion is slow and at the same time kong A sharp decline in performance .

After several tests, we determined 「 Just call admin The interface of leads to kong Performance degradation 」 This conclusion , It is very similar to the problem of the community , Links are as follows ：

https://github.com/Kong/kong/issues/7543

kong Instance read-write separation

I'm sure it's admin After the reason of the interface , We decided to admin Business related kong Instance separation , hope admin The call of will not affect the normal traffic access of the business , Expect to achieve kong Of admin Slow interface , But don't affect the access performance of the business .

However , No effect .

postgres transfer RDS

kong After the efforts at the level are fruitless , We also observed when calling admin Interface test ,postgres The process of has also increased a lot ,CPU The utilization rate has also increased , Also decided to pg Migrate to More professional RDS in .

still , No effect .

Roll back

Finally, we rolled back to 0.14 edition , Pursue temporary “ Peace of mind ”.

thus , The online attempt is basically a paragraph , It also roughly finds out the conditions for the recurrence of the problem , So we decided to build an environment offline to continue to find out the cause of the problem .

The way to reproduce the problem

We will have problems kong Of postgres Import a copy of the data into the development environment , simulation 「 call admin Interface is a sharp decline in performance 」 The situation of , And find a solution .

Import data

We will have problems in the cluster postgre After the data is backed up, it is imported into a new cluster :

psql -h 127.0.0.1 -U kong < kong.sql

Copy

And turn it on kong Of prometheus plug-in unit , Easy to use grafana To view the performance icon :

curl -X POST http://10.97.4.116:8001/plugins --data "name=prometheus"

Copy

Phenomenon one

call admin Service /   Same slow , It is consistent with the online phenomenon , Call when the amount of data is large admin Of / Directories take more time .

curl http://10.97.4.116:8001

Copy

Phenomenon two

Then let's simulate the call encountered online admin Poor service access performance after interface , First call admin Interface to create a business api, For the test , We created one service And one. routes：

curl -i -X POST http://10.97.4.116:8001/services/ -d 'name=baidu2'  -d 'url=http://www.baidu.com'
curl -i -X POST http://10.97.4.116:8001/services/baidu2/routes \-d "name=test2" \     -d "paths[1]=/baidu2"

Copy

You can use it later  curl http://10.97.4.116:8000/baidu2 To simulate the business interface for testing .

Get ready admin Interface test script , Create and delete a service/route, Intersperse one in the middle service list.

#!/bin/bash

curl -i -X POST http://10.97.4.116:8001/services/ -d 'name=baidu'  -d 'url=http://www.baidu.com'

curl -i -X POST http://10.97.4.116:8001/services/baidu/routes \
     -d "name=test" \
     -d "paths[1]=/baidu" 

curl -s http://10.97.4.116:8001/services

curl -i -X DELETE      http://10.97.4.116:8001/services/baidu/routes/test

Copy

Then continue to call the script ：

for i in `seq 1 100`; do sh 1.sh ; done

Copy

In the process of continuously calling the script, access a business interface , You will find it very slow , It is completely consistent with the online phenomenon .

curl  http://10.97.4.116:8000/baidu2

Copy

PS： Streamline scripts , Only one write is triggered after , Or deletion will also trigger this phenomenon

Accompanying phenomenon

• kong Example of cpu Follow mem Both continue to rise , And when admin This phenomenon is still not over after the interface call .mem It will rise to a certain extent nginx worker process oom fall , And then restart , This may be the reason for the slow access ;
• We set it up KONG_NGINX_WORKER_PROCESSES by 4, And for pod The memory of is 4G When ,pod The overall memory will be stable at 2.3G, however   call admin Interface test ,pod Memory will keep rising to more than 4G, Trigger worker Of OOM, So I will pod The memory of is adjusted to 8G. Call again admin Interface , Find out pod Memory is still rising , It just rose to 4.11 G It's over , This seems to mean that we are going to set pod   The memory of is KONG_NGINX_WORKER_PROCESSES twice as much , This problem is solved （ But there is another important question is why to call once admin Interface , It will cause the memory to rise so much ）;
• in addition , When I keep calling admin At the interface , The final memory will continue to grow and stabilize to 6.9G.

At this time, we will abstract the problem ：

call 「kong admin Interface 」 Cause the memory to keep rising , And then trigger oom Lead to worker By kill fall , Eventually, business access is slow .

Continue to investigate what is taking up memory ：

I use pmap -x [pid]   I checked it twice worker Memory distribution of the process , What changes is the part framed in the second picture , Judging from the address, the whole memory has been changed , But after exporting and stringing the memory data , There is no effective information for further investigation .

Conclusion

• The question is related to kong The upgrade （0.14  -->  2.2.0） It doesn't matter. , Use it directly 2.2.0 Version will also have this problem ;
• kong every other worker_state_update_frequency It will be rebuilt in memory after time router, Once reconstruction starts, it will lead to Memory goes up , After looking at the code, the problem is Router.new Here's the way , Will apply for lrucache But there is no flush_all, According to the latest 2.8.1 Version of lrucache After the release, the problem still exists ;
• That is to say kong Of Router.new When other logic in the method arrives, the memory rises ;

• This shows that the problem is kong There is a performance bug, It still exists in the latest version , When route Follow service When reaching a certain order of magnitude, there will be calls admin Interface , Lead to kong Of worker Memory is rising rapidly , bring oom This leads to poor business access performance , The temporary solution can be to reduce NGINX_WORKER_PROCESSES And increase kong pod Of memory , Make sure to call admin The memory required after the interface is enough to use without triggering oom, To ensure the normal use of business .

Last , We will be in https://github.com/Kong/kong/issues/7543 This issue Add this phenomenon , You are welcome to continue to pay attention , Discuss together ～

---