Redis problem (I) -- cache penetration, breakdown, avalanche

Cache penetration

key The corresponding data does not exist in the data source , Each time for times key The request for could not be fetched from the cache , Requests are pushed to the data source , That could crush the data source .

For example, use a non-existent user id To get user information , There is no such data in the cache or database , Hackers may use this vulnerability to attack and overwhelm the database

The goal is to crash the server

What happened

1. The pressure on the application server has increased
2. redis Hit rate reduced
3. Keep querying the database

The reason for this

1. redis No data in , There's nothing in the database
2. There are a lot of anomalies url visit

Solution

There must be no data that cannot be cached or queried , Because the cache is written passively on Miss , And for the sake of fault tolerance , If the data cannot be queried from the storage tier, the cache will not be written , This will cause the non-existent data to be queried by the storage layer every time it is requested , It loses the meaning of caching .

1. Cache null values ： If the data returned by a query is empty （ Whether or not the data does not exist ）, We still cache this empty result , Setting the expiration time for empty results can be very short , Not more than 5 minute
2. Set up an accessible list （ White list ）： Use bitmaps Type defines a list of accessible , list id As bitmaps The offset , Every visit and bitmap Medium id Compare , If you visit id be not in bitmaps in , To intercept , Access not allowed .
3. Use of Blum filter ： The bloom filter is actually a very long binary vector （ Bitmap ） And a series of random mapping functions （hash function ）. The bloom filter can retrieve whether an element is in a collection . Its advantage is that the space efficiency and query time are far more than the general algorithm , The disadvantage is that it has certain error recognition rate and deletion difficulty
4. Real-time monitoring ： If I found redis It's starting to drop , Need to check access objects and data , Cooperate with operation and maintenance personnel , You can set up a blacklist to restrict services

Cache breakdown

key The corresponding data exists , But in redis Medium overdue , At this time, if a large number of concurrent requests come , These requests are read from the database and set back to the cache , At this time, a large amount of concurrency may overwhelm the database .

What happened

1. Database access pressure increases instantaneously
2. redis Not in a large number key Be overdue
3. redis The normal operation

The reason for this

1. redis One of them key Out of date , At the same time the key Be visited a lot

Solution

key It may be accessed at some point in time with super high concurrency , Is a very popular data . At this time, we need to consider a problem ： The problem of cache breakdown .

1. Preset hot data ： stay redis Before the peak visit , Put some hot data into redis in , Increase these hot key Duration
2. Timely adjustment ： On site monitoring of those data is popular , Adjust expiration time in real time
3. Using locks ：
   1. When the cache fails （ The value is empty ）, Not immediately load db
   2. First, use the caching tool to perform some operations with the return value of successful operations （ such as setnx）

Cache avalanche

key The corresponding data exists , But in redis Medium overdue , At this time, if a large number of concurrent requests come , These requests are read from the database and set back to the cache , At this time, a large amount of concurrency may overwhelm the database .

The difference between cache avalanche and cache breakdown is cache avalanche For many key cache .

What happened

The pressure on the database increases and the server crashes

The reason for this

In a very small period of time , A lot of key The centralized expiration of

terms of settlement

1. Building a multi-level cache architecture ：nginx cache +redis cache + Other caches
2. Use locks or queues ： Lock or queue is used to ensure that there will not be a large number of threads reading and writing to the database at one time , So as to avoid a large number of concurrent requests falling on the underlying storage system in case of failure . Not for high concurrency