Redis introduction and Practice (with source code)

 

Continue to update today austin project , If you haven't seen this series, you can click my history article to review , In the process of watching, don't forget give the thumbs-up yo ！ It is recommended not to miss or jump to see , Otherwise, I won't understand this article , I won't repeat the knowledge points and business I wrote before . Programming learning materials, click to get free

What we want to achieve today is handler After consumption news , Realization Platform De duplication function .

01、 What is de duplication and idempotent

I was on this topic before 《 On line interviewers 》 The series has been shared , This interview will often ask , I can review with you again

「 idempotent 」 and 「 duplicate removal 」 The essence of ：「 only Key」+「 Storage 」

only Key How to build and choose what storage to use , It's all business decisions .「 Local cache 」 If the business is appropriate , It can be used as 「 In front of 」 Sift out a part of , Use other storage as 「 After 」, Use this mode to improve performance .

What I want to talk about today Redis, It has High performance reading and writing , Both pre screening and post judgment can ,austin The de duplication function of the project depends on Redis The implementation of the .

02、 install REDIS

Let's go through it quickly Redis Use your posture （ If you are not interested in this, you can jump directly to 05 Explain the relevant business and code design ）

install Redis The environment is the same as last time Kafka It's the same , For convenience, I'll continue to use docker-compose In the way of .

Environmental Science ：CentOS 7.6 64bit

First , Let's create a new folder redis, Then create... In this directory data Folder 、redis.conf Document and docker-compose.yaml file

redis.conf The contents of the document are as follows ( The following configuration can be changed here , such as requirepass The password I specified is austin)

protected-mode no
port 6379
timeout 0
save 900 1 
save 300 10
save 60 10000
rdbcompression yes
dbfilename dump.rdb
dir /data
appendonly yes
appendfsync everysec
requirepass austin
 Copy code 

docker-compose.yaml The contents of the document are as follows ：

version: '3'
services:
  redis:
    image: redis:latest
    container_name: redis
    restart: always
    ports:
      - 6379:6379
    volumes:
      - ./redis.conf:/usr/local/etc/redis/redis.conf:rw
      - ./data:/data:rw
    command:
      /bin/bash -c "redis-server /usr/local/etc/redis/redis.conf "
 Copy code 

The configuration work is finished , If it's a cloud server , Remember to drive redis port 6379

03、 start-up REDIS

start-up Redis Same as before Kafka It's about the same time

docker-compose up -d
​
docker ps
​
docker exec -it redis redis-cli
 Copy code 

Get into redis After the client , We want to see if it's normal .（ Before formally entering the order , We need to pass password verification , The password configured under the configuration file is austin）

Then feel free to see if the command is normal OK La

04、JAVA Use in REDIS

stay SpringBoot In the environment , Use Redis It's very simple （ Once again, it reflects the use of SpringBoot The benefits of ）. All we need to do is pom Import the corresponding dependency under the file , And configure... Under the configuration file host/port and password That's all. .

For clients , We'll just use it RedisTemplate Just fine , It is a high degree of encapsulation of the client , It's already working well .

05、 De functional business

Any function code implementation is inseparable from the business scenario , Before talking about code implementation , Business first ！ Usually when doing demand , I've always believed in ： First understand what the business needs to do , Then realize the function .

The function of de duplication is in austin In the project, I positioned it as ： Platform functionality . It's important to understand this ！ Don't think about doing all kinds of de duplication logic of the business on the platform , This is not reasonable .

This can only be Generality The de duplication function of... Has been done , Follow Business is strongly linked It should be realized by the business party itself . therefore , What I have achieved here is ：

• 5 If the same user receives the same content within minutes , Should be filtered out . Reason for realization ： Probably because MQ Repeated consumption Or is it The business party calls carelessly , Cause the same message in In a short time By austin consumption , Then send it to the user . It's time to go heavy , We can To some extent Reduce accidents .
• If the same user has received content from a certain channel within one day 5 Time , Should be filtered out . Reason for realization ： Under the operation or business push , It is possible that some users will receive push messages many times in a day . Avoid too much interruption to users , According to the general rules, users can only receive N Bar message .

With the development of business , There are also some de duplication functions that we need to do , But remember , We don't deal with business here Strong hook .

When our core functions depend on other middleware , We try to Avoid that the core functions cannot be used normally due to the abnormality of middleware . such as ,redis If you hang up , Nor should it affect the distribution of our normal messages , It can only affect the function of weight removal .

06、 Overview of de duplication function code

Before , We have Kafka After pulling the news , Then put the message into their own thread pool for processing , For the function of de duplication, we only need to send it before .

I'll go back to the logic Unified abstraction by ： stay X Period of time Neda arrived Y threshold . The steps of de duplication can be simply divided into ：

• from Redis Get records
• Judge Redis Whether the existing records meet the conditions
• If the conditions are met, the weight will be removed , If it does not meet the conditions, re insert Redis

What I'm using here is Template method pattern ,deduplication Methods have been defined to locate , When a new de duplication logic needs to be accessed , Just inherit AbstractDeduplicationService To achieve deduplicationSingleKey The method can .

such as , I take the de duplication logic of sending the same content to the same user as an example ：

07、 Specific implementation of de duplication code

In this scene , I use Redis It's all used Batch Operations to reduce requests Redis The number of times , For our business scenario （ When consuming, you need a lot of requests Redis, Use The batch operation The improvement is still great ）

Because I think there are many scenes to use , So I encapsulated a RedisUtils Tool class , And what you can find is ： I'm interested in the operation Redis All places use try catch Come and wrap it . even Redis Malfunction , My core business will not be affected .

08、 Your code does BUG！

I wonder if you can see the problem after reading the above code , There are some handsome people who like to praise We can see two problems directly ：

• Why is your de duplication function done before sending messages ？ What if you fail to send a message ？
• There is a concurrency problem with your de duplication function ？ Suppose I have two identical messages , There are multiple threads for consumption , Then the two threads query Redis, Found that they were not Redis Inside , Then there is the problem of concurrency

you 're right , The above two problems exist . however , I'm not going to solve it on my side .

Let's look at the first question ：

For this question , There are two reasons I can argue ：

• Suppose I fail to send a message , In this system, there will be no backtracking MQ Way to resend the message （ to flash back MQ The impact of re consumption is too great ）. We can send the failed userId Write it down （ The related log system will be improved later ）, With userId in the future , We Manually resend in batches Just fine . There is no need for the business party to call the interface manually , Directly through similar excel Just import it in the same way .
• In business , Many scenarios of sending messages Even if you really lose a few pieces of data , Will not be found . Some news is very important , But more information is not so important , And we Even when the interface is called, the data is written to Redis, But messages from many channels are actually sent after calling the interface , I don't know if it's really sent to the user .

Let's look at the second question ：

If we're going to just Redis To realize the function of de duplication , Want to have no concurrency problem at all , That's worth it lua Script , But up lua Scripts are costly . The implementation of de duplication depends on two operations ： Query and insert . If there is no... After query , You need to add . That query and insert need to keep Atomicity To avoid the problem of concurrency

Then bring the perspective back to us Why should we realize the de duplication function ：

When there is an accident , We can ensure that most Your message will not be repeated . For the rules of integrity , Rules are broken due to concurrent message sending The probability of is very low .

09、 summary

This article briefly describes Redis Installation of and in SpringBoot How to use Redis, It mainly explains why to realize the function of de duplication, as well as the design of code and the specific implementation of function .

Technology is inseparable from business , It is possible that the code we design or implement is for Strong consistency There are omissions , But if the whole system is simpler and more efficient , When the business is acceptable , It's not impossible .

This is a kind of trade-off Balance , To ensure that data is not lost or repeated, it is generally necessary to Write more code and Loss of system performance Wait to get it . I can do this when I consume messages at least once semantics , Ensure that data is not lost . I can consume news , Achieve true idempotence , The downstream call will not repeat .

But these are Conditional Of , To achieve at least once semantics , Manual required ack. To achieve idempotence , Need to use redis lua Or write the record to MySQL Build unique key And put the key Set unique index . It is necessary in the scenario of order , But in a core messaging system , Maybe it's not that important .

①3000 Multiple copies Python E-books have
②Python Development environment installation tutorial has
③Python400 The self-study video has
④ The common vocabulary of software development is
⑤Python The learning roadmap has
⑥ Project source code case sharing has

If you can use it, you can take it , In my QQ In the technology exchange group （ Technical exchange and resource sharing , No advertising ） You can take it by yourself , Group number is 895937462.