Performance problems caused by redis cache invalidation and competitive loading

redis When the cache fails , as everyone knows , If you don't protect yourself, you can allow requests to reload the cache concurrently , It will cause a large number of requests directly to the database , That is, cache breakdown occurs .

distcache To solve this problem , It uses the method of contention to obtain lock and refresh cache , Deal with cache invalidation .

such as A B Two requests arrive , stay redis Compete for locks , The thread that obtains the lock starts to look up the table and brush the cache . The thread that cannot acquire the lock starts to pass through the loop sleep The way , Query whether the cache has been flushed , Spin at regular intervals , If the cycle waiting time exceeds a certain value , Just give up .

It looks perfect , Actually caused Redis Performance problems of

mas In production environment , Find out redis Both the request delay and the request volume are in 2 There are periodic peaks around hours .

mas The number of servers in the production environment is about 200 , To single slot Interface request for qps The peak is about 6000 , The actual time required to reload the cache is not short , If it exceeds 1 second , Theoretically, it will bring 6000*20 Of redis single key qps, Already exceeded redis Maximum single node performance .

The inspiration of this question ： Don't enlarge the request in the request .