Redis - detailed explanation of cache avalanche, cache penetration and cache breakdown

Cache avalanche 、 Cache penetration 、 Cache breakdown

---

1. Cache avalanche

1.1 Under what circumstances

1. Redis The mainframe is down ,Redis Total collapse
2. A large amount of data in the cache is out of date at the same time

1.2 terms of settlement

1. redis Cache clusters achieve high availability （ In advance ）
2. ehcache Local cache + Hystrix Or Ali sentinel Current limiting & Downgrade （ In the matter ）
3. Turn on Redis Persistence mechanism aof/rdb, Recover the cache cluster as soon as possible （ After the event ）
   

2. Cache penetration

1. Request to query a record , First redis after mysql It is found that the record cannot be queried , But every time the request is typed on the database , This leads to a surge in the pressure on the background database , This phenomenon is called cache penetration , This redis Into a decoration ...
2. Simply put, there is nothing , Neither in Redis In cache , It's not in the database

2.1 harm

1. After the first query , Usually we write back redis Mechanism
2. The second time I came to check redis And then there is , Occasional penetration is generally irrelevant

2.2 solve

Solution 1： Empty object cache or default value

commonly OK

But
Hackers or malicious attacks

1. Hackers will attack your system , Take a nonexistent id Go look up the data , A large number of requests will be generated to query the database . It may cause your database to crash due to too much pressure
2. id The same hit your system ： The first time I hit mysql, After the empty object is cached, it returns... The second time null 了 , avoid mysql Be attacked , No more walking around the database
3. id Different systems （ I can't help it ）： Due to the existence of empty object cache and cache writeback ( Look at your business, not limited to death ),redis Irrelevant in key And I'll write more and more ( Remember to set up redis Expiration time )

Solution 2：Google The bloon filter Guava Solve cache penetration

Features of bloon filter ： Misjudgment will only occur in elements that have not been added to the filter , There will be no misjudgment for the added elements

Guava The implementation of CBF is quite authoritative , So in the actual project, we don't need to implement a bloom filter manually

• Code combat

public class GuavaBloomfilterDemo {
    
    public static final int _1W = 10000;
    // How much data is expected to be inserted into the bloom filter 
    public static int size = 100 * _1W;
    // Miscalculation rate , The smaller it is, the less the number of misjudgments ( reflection , Is it possible to set the infinitesimal , Wouldn't it be better not to misjudge )
    public static double fpp = 0.01; // The error rate is particularly small , The efficiency of program execution drops sharply 

    /** * helloworld introduction  */
    public void bloomFilter() {
    
        //  Create a bloom filter object 
        BloomFilter<Integer> filter = BloomFilter.create(Funnels.integerFunnel(), 100);
        //  Determines whether the specified element exists 
        System.out.println(filter.mightContain(1));//false
        System.out.println(filter.mightContain(2));//false
        //  Add elements to the bloom filter 
        filter.put(1);
        filter.put(2);
        System.out.println(filter.mightContain(1));//true
        System.out.println(filter.mightContain(2));//true

    }

    /** *  False positive rate demonstration + Source code analysis  */
    public void bloomFilter2() {
    
        //  Build bloom filter 
        BloomFilter<Integer> bloomFilter = BloomFilter.create(Funnels.integerFunnel(), size, fpp);

        //1  First insert... Into the bloom filter 100 Million sample data 
        for (int i = 0; i < size; i++) {
    
            bloomFilter.put(i);
        }
   /* List<Integer> listSample = new ArrayList<>(size); //2  this 100 Million sample data , Whether they all exist in the bloom filter ？ for (int i = 0; i < size; i++){ if (bloomFilter.mightContain(i)) { listSample.add(i); continue; } } System.out.println(" Number of existing ：" + listSample.size());// Number of existing ：1000000 */

        //3  Deliberately take 10 Ten thousand values that are not in the filter , See how many are thought to be in the filter , False positive rate demonstration 
        List<Integer> list = new ArrayList<>(10 * _1W);

        for (int i = size + 1; i < size + 100000; i++) {
    
            if (bloomFilter.mightContain(i)) {
    
                System.out.println(i + "\t" + " Misjudged .");
                list.add(i);
            }
        }
        System.out.println(" The number of misjudgments ：" + list.size()); // The number of misjudgments ：947
    }

    public static void main(String[] args) {
    
        new GuavaBloomfilterDemo().bloomFilter2();
    }
}

Solution 3：Redis Bloom filter solves cache penetration

Guava The disadvantages are explained ： Only for single use

Case study ： White List Filter

1. Architecture diagram
   
2. Code

public class RedissonBloomFilterDemo {
    
    public static final int _1W = 10000;

    // How much data is expected to be inserted into the bloom filter 
    public static int size = 100 * _1W;
    // Miscalculation rate , The smaller it is, the less the number of misjudgments 
    public static double fpp = 0.03;

    static RedissonClient redissonClient = null;//jedis
    static RBloomFilter rBloomFilter = null;//redis The built-in bloom filter 

    @Resource
    RedisTemplate redisTemplate;


    static {
    
        Config config = new Config();
        config.useSingleServer().setAddress("redis://192.168.111.147:6379").setDatabase(0);
        // structure redisson
        redissonClient = Redisson.create(config);
        // adopt redisson structure rBloomFilter
        rBloomFilter = redissonClient.getBloomFilter("phoneListBloomFilter", new StringCodec());

        rBloomFilter.tryInit(size, fpp);

        // 1 test   The bloom filter has +redis Yes 
        //rBloomFilter.add("10086");
        //redissonClient.getBucket("10086",new StringCodec()).set("chinamobile10086");

        // 2 test   The bloom filter has +redis nothing 
        //rBloomFilter.add("10087");

        //3  test  , The bloom filter has no +redis nothing 

    }

    private static String getPhoneListById(String IDNumber) {
    
        String result = null;

        if (IDNumber == null) {
    
            return null;
        }
        //1  First go to the bloom filter to check 
        if (rBloomFilter.contains(IDNumber)) {
    
            //2  There is... In the bloom filter , Go again redis Look inside 
            RBucket<String> rBucket = redissonClient.getBucket(IDNumber, new StringCodec());
            result = rBucket.get();
            if (result != null) {
    
                return "i come from redis: " + result;
            } else {
    
                result = getPhoneListByMySQL(IDNumber);
                if (result == null) {
    
                    return null;
                }
                //  Update the data back to redis
                redissonClient.getBucket(IDNumber, new StringCodec()).set(result);
            }
            return "i come from mysql: " + result;
        }
        return result;
    }

    private static String getPhoneListByMySQL(String IDNumber) {
    
        return "chinamobile" + IDNumber;
    }


    public static void main(String[] args) {
    
        //String phoneListById = getPhoneListById("10086");
        //String phoneListById = getPhoneListById("10087"); // Please test execution 2 Time 
        String phoneListById = getPhoneListById("10088");
        System.out.println("------ The result of the search ： " + phoneListById);

        // Pause the thread for a few seconds 
        try {
    
            TimeUnit.SECONDS.sleep(1);
        } catch (InterruptedException e) {
    
            e.printStackTrace();
        }
        redissonClient.shutdown();
    }

}

stay centos7 Lower bloom filter 2 Installation methods

1. use docker install RedisBloom, recommend
   Redis stay 4.0 Then there are plug-in functions （Module）, You can use external extensions , have access to RedisBloom As Redis Brom filter insert .

    docker run -p 6379:6379 --name=redis6379bloom -d redislabs/rebloom  
	docker exec -it redis6379bloom /bin/bash        
	redis-cli

Common operation commands of Bloom filter

3. Cache breakdown

A large number of requests query one at a time key when , At this point key It just failed , It will result in a large number of requests to the database
In short, it is hotspot key Sudden failure 了 , Suddenly and violently dozen mysql

3.1 harm

It will cause too many database requests at a certain time , The pressure is building up .

3.2 solve

programme 1： For frequently accessed hotspots key, Just don't set the expiration time

programme 2： Mutually exclusive locks prevent breakdown

Multiple threads query this data in the database at the same time , Then we can query data in the first request Use a mutex to lock it .
Other threads go to this point and wait until they get the lock , Wait for the first thread to find the data , Then do caching . Later threads come in and find that there is already a cache , Just go to cache .

programme 3： Regularly polling , Mutually exclusive updates , Differential failure time

Case study ： Taobao poly cost-effective function to achieve + Prevent cache breakdown

Highly concurrent Taobao gathering is cost-effective Case landing （Redis Of list）

@Service
@Slf4j
public class JHSABTaskService {
    
    @Autowired
    private RedisTemplate redisTemplate;

    //@PostConstruct , Usual notes , Open with time 
    public void initJHSAB() {
    
        log.info(" start-up AB Timer planning task Taobao poly cost-effective function simulation .........." + DateUtil.now());
        new Thread(() -> {
    
            // Analog timer , Regularly put the special offers in the database , Refresh to redis in 
            while (true) {
    
                // Simulate reading from the database 100 A special offer , Used to load into a cost-effective page 
                List<Product> list = this.products();
                // To update B cache 
                this.redisTemplate.delete(Constants.JHS_KEY_B);
                this.redisTemplate.opsForList().leftPushAll(Constants.JHS_KEY_B, list);
                this.redisTemplate.expire(Constants.JHS_KEY_B, 20L, TimeUnit.DAYS);
                // Update again A cache 
                this.redisTemplate.delete(Constants.JHS_KEY_A);
                this.redisTemplate.opsForList().leftPushAll(Constants.JHS_KEY_A, list);
                this.redisTemplate.expire(Constants.JHS_KEY_A, 15L, TimeUnit.DAYS);
                // One minute apart   Do it again 
                try {
    
                    TimeUnit.MINUTES.sleep(1);
                } catch (InterruptedException e) {
    
                    e.printStackTrace();
                }

                log.info("runJhs Refresh regularly ..............");
            }
        }, "t1").start();
    }

    /** *  Simulate reading from the database 100 A special offer , Used to load into a cost-effective page  */
    public List<Product> products() {
    
        List<Product> list = new ArrayList<>();
        for (int i = 1; i <= 20; i++) {
    
            Random rand = new Random();
            int id = rand.nextInt(10000);
            Product obj = new Product((long) id, "product" + i, i, "detail");
            list.add(obj);
        }
        return list;
    }
}

@RestController
@Slf4j
@Api(description = " Poly cost-effective product list interface AB")
public class JHSABProductController {
    
    @Autowired
    private RedisTemplate redisTemplate;

    @RequestMapping(value = "/pruduct/findab", method = RequestMethod.GET)
    @ApiOperation(" Display capacity by page and per page , Click to see AB")
    public List<Product> findAB(int page, int size) {
    
        List<Product> list = null;
        long start = (page - 1) * size;
        long end = start + size - 1;
        try {
    
            // use redis list data-structured lrange Command to realize paging query 
            list = this.redisTemplate.opsForList().range(Constants.JHS_KEY_A, start, end);
            if (CollectionUtils.isEmpty(list)) {
    
                log.info("=========A The cache has expired , Remember to repair it manually ,B Cache auto continuation 5 God ");
                // The user queries the cache first A( The above code ), If caching A No queries （ for example , Deleted when updating the cache ）, Then query the cache B
                this.redisTemplate.opsForList().range(Constants.JHS_KEY_B, start, end);
            }
            log.info(" Query results ：{}", list);
        } catch (Exception ex) {
    
            // The exception here , It's usually redis paralysis  , or  redis The Internet timeout
            log.error("exception:", ex);
            //TODO  go DB Inquire about 
        }
        return list;
    }
}

summary