Redis can only cache? Too out!

Most databases , Because I often deal with disks , In high concurrency scenarios , The response will be very slow . In order to solve this speed difference , Most systems habitually add a cache layer , To speed up data reading .redis Because of its excellent processing power and rich data structure , It has become a de facto distributed caching standard .

however , If you think redis If you can only cache , Then it's too small .

redis Rich data structure , This makes its business usage scenarios very extensive , add rdb The persistence feature of , It can even be used as a landing database . under these circumstances ,redis Can support most Internet companies , Especially social 、 game 、 Half of the live broadcasting companies .

 

1、Redis Capable of storage

redis It provides a very rich cluster mode ： Master-slave 、 sentry 、cluster, Meet the demand for high availability of services . meanwhile ,redis There are two ways to persist ：aof and rdb, What is commonly used is rdb.

adopt bgsave Instructions , The main process will fork A new process , Write back to disk .bgsave It's equivalent to taking a snapshot , Because it does not WAL Journal and checkpoint Mechanism , You can't do real-time backup . If the machine suddenly loses power , Then it's easy to lose data .

Fortunately, ,redis It's a memory database , The speed of main plex synchronization is very fast . If your cluster is well maintained , Reasonable memory allocation , Then unless the machine room is powered off , otherwise redis Of SLA, Will remain at a very high level .

It doesn't sound absolutely reliable , There is a possibility of losing data ！ This is in general CRUD The business of , It's intolerable . What is it for? redis It can meet the needs of most Internet companies ？ This is also determined by business attributes .

In deciding to embrace as much as possible redis Before , You need to confirm whether your business has the following characteristics ：

In addition to the core business , Whether most businesses have low requirements for data reliability , Losing one or two pieces of data is tolerable ？

Faced with C End user , According to the user ID Quickly locate a class of data , Data sets are generally small ？ There is no need for a large number of range queries ？

Whether it can bear the cost demand of memory data ？

Whether the business requires almost no transaction operations ？

Fortunately , There are many special needs for this kind of business . For example, common social , game 、 live broadcast 、 Operational business , Can be completely dependent on Redis Of .

 

2. Reids Application scenarios

Redis With loose document structure , Rich data types , Able to adapt to ever-changing scheme Change requirements , I'm going to talk about that Redis A large number of application scenarios other than caching .

2.1 Basic user data storage

In traditional database design , User tables are very difficult to design , When changing, it will hurt muscles and bones . Use Redis Of hash structure , Loose data model design can be realized . Some are not fixed , Functional properties of verification type , We can use JSON The interface is stored directly in hash Of value in . Use hash structure , May adopt HGET and HMGET Such as instruction , Only get the data you need , It is also very convenient to use .

>HSET user:199929 sex m>HSET user:199929 age 22>HGETALL user:1999291) "sex"2) "m"3) "age"4) "22"

This non statistical 、 Read more and write less , It is very suitable for KV Structure for storage .Redis Of hash Structure provides a very rich set of instructions , You can also use HINCRBY Increase and decrease , Very convenient .

2.2 Implementation counter

It mentioned a little HINCRBY Instructions , And for Redis Of Key itself , Also have INCRBY Instructions , Realize the increment and decrement of a value .

For example, the following scenarios ： Count the likes of a post ; Store the number of concerns about a topic ; Number of fans storing a tag ; Store a general number of comments ; The popularity of a post ; The number of red dot messages ; give the thumbs-up 、 like 、 Collection number, etc .

> INCRBY feed:e3kk38j4kl:like 1> INCRBY feed:e3kk38j4kl:like 1> GET feed:e3kk38j4kl:like"2"

A hot business like microblog , Traditional database , It must not hold , With the help of an in memory database . because Redis Very fast , You don't have to use tradition DB Very slow count operation , All such incremental operations are on the millisecond level , And the effect is real-time .

2.3 Ranking List

Leaderboards can improve the enthusiasm of participants , So this business is very common , It's essentially a topn The problem of .

Redis One of them is called zset Data structure of , An ordered list implemented using a jump table , You can easily implement problems such as leaderboards . When the zset Data in , Reach the level of tens of millions or even billions , Can still maintain very high concurrent read and write , And has a great average response time （5ms within ）.

Use zadd You can add new records , We will use ranking related scores , As a record score value , And then use zrevrange Command to obtain real-time leaderboard data , and zrevrank You can easily get the real-time ranking of users .

>ZADD sorted:xjjdog:2021-07  55 dog0>ZADD sorted:xjjdog:2021-07  89 dog1>ZADD sorted:xjjdog:2021-07  32 dog2>ZCARD sorted:xjjdog:2021-07>3> ZREVRANGE  sorted:xjjdog:2021-07  0 -10 WITHSCORES # top10 Ranking List 1) "dog1"2) "89"3) "dog0"4) "55"5) "dog2"6) "32"

2.4 Friend relationship

set structure , Is a collection without duplicate data , You can put a user's attention list 、 Fans list 、 Two way follow list 、 The blacklist 、 Like list, etc , Use independent zset For storage .

Use ZADD、ZRANK etc. , Use the user's blacklist ZADD add to ,ZRANK Use the returned sorce Value to determine whether it is in the blacklist . Use sinter Instructions , Can get A and B Our mutual friends .

In addition to friends , With a clear blacklist 、 White list business scenario data , You can use set Structure for storage . There are many other business scenarios , For example, the address book uploaded by a user , Calculate the friend relationship of the address book, etc .

In practical use , Use zset Store more of these relationships .zset Same as set equally , Duplicate values are not allowed , but zset One more. score Field , We can store a timestamp , Used to indicate when the relationship was established , Have a clearer business meaning .

2.5 Count the number of active users

Similar statistics of daily active users 、 User check-in 、 User online status , This scattered demand , There are too many . If you store one for each user bool Variable , That takes up too much space . In this case , We can use bitmap structure , To save a lot of storage space .

>SETBIT online:2021-07-23 3876520333 1>SETBIT online:2021-07-24 3876520333 1>GETBIT online:2021-07-23 38765203331>BITOP AND active online:2021-07-23 online:2021-07-24>GETBIT active 38765203331>DEBUG OBJECT online:2021-07-23Value at:0x7fdfde438bf0 refcount:1 encoding:raw serializedlength:5506446 lru:16410558 lru_seconds_idle:5(0.96s)

Be careful , If your id It's big , You need to do a preprocessing first , Otherwise it will take up a lot of memory .

bitmap Contains a continuous string of 2 Hexadecimal Numbers , Use 1bit To express the question of true and false . stay bitmap On , have access to and、or、xor Equipotential operation (bitop).

2.6 Distributed lock

Redis Distributed locks for , Is a lightweight solution . Although it's not as reliable as Zookeeper Systems like that , but Redis Distributed locks have very high throughput .

One of the most rudimentary locking actions , have access to redis belt nx and px Parametric set Command to complete . Here is a simple distributed sample code .

public String lock(String key, int timeOutSecond) {    for (; ; ) {        String stamp = String.valueOf(System.nanoTime());        boolean exist = redisTemplate.opsForValue().setIfAbsent(key, stamp, timeOutSecond, TimeUnit.SECONDS);        if (exist) {            return stamp;        }    }}public void unlock(String key, String stamp) {    redisTemplate.execute(script, Arrays.asList(key), stamp);}

Delete the lua by .

local stamp = ARGV[1]local key = KEYS[1]local current = redis.call("GET",key)if stamp == current then    redis.call("DEL",key)    return "OK"end

redisson Of RedLock, Is the most widely used distributed lock solution , There is a difference between read and write locks , And handled many redis Exception problem in case of instance .

2.7 Distributed current limitation

Use counters to achieve simple current limiting , stay Redis It is very convenient in , Just use incr coordination expire Command can .

 incr key expire key 1

This simple implementation , Usually there's no problem , But when the flow is relatively large , There is a risk of a sudden surge in traffic over a time span . The root cause , It's this way of time segmentation. It's too fixed , There is no smooth transition scheme like sliding window .

The same is redisson Of RRateLimiter, Realized and guava A similar distributed current limiting tool class in , It's very convenient to use . Here's a short example ：

RRateLimiter limiter = redisson.getRateLimiter("xjjdogLimiter"); //  It only needs to be initialized once  //  Every time 2 Second 5 Permission  limiter.trySetRate(RateType.OVERALL, 5, 2, RateIntervalUnit.SECONDS);  //  No license available , Will be blocked all the time      limiter.acquire(3);

2.8 Message queue

redis Simple queues can be implemented . On the producer side , Use LPUSH Add to a list ; On the consumer side , Continuous use RPOP Instruction to fetch the data , Or use blocked BRPOP Command to get data , Suitable for small-scale rush buying demand .

Redis also PUB/SUB Pattern , however pubsub It is more suitable for business such as message broadcasting .

stay Redis5.0 in , Added stream Type of data structure . It is more similar to Kafka, There is the concept of theme and consumption group , Multicast and persistence can be realized , Has been able to meet most business needs .

2.9 LBS application

Early in Redis3.2 edition , It's launched GEO function . adopt GEOADD Command append lat、lng Latitude and longitude data , The distance between coordinates can be calculated 、 Include relationship calculations 、 Functions such as people nearby .

About GEO function , The most powerful open source solution is based on PostgreSQL Of PostGIS, But for a general scale GEO service ,redis It's enough .

2.10 More extended application scenarios

Want to see redis Can do , I have to mention the following java Client class library redisson.redisson Contains rich distributed data structures , All based on redis Designed .

redisson For example Set、 SetMultimap、 ScoredSortedSet、 SortedSet, Map、 ConcurrentMap、 List、 ListMultimap、 Queue、BlockingQueue And so on , Make based on redis Programming is more convenient . stay github On , You can see that there are hundreds of such data structures ：https://github.com/redisson/redisson/tree/master/redisson/src/main/java/org/redisson/api.

For a language , Basic array 、 Linked list 、 Collection etc. api, Together, it can complete the development of most businesses .Redis No exception , It has these basic api Operational capability , Can also combine the composition of cloth 、 Thread safe high concurrency applications .

because Redis It's memory based , So it's very fast , We will also use it as a storage place for intermediate data . For example, some common configurations , Put it in redis Share in , It acts as a configuration center ; For example JWT Store your token in Redis in , You can break through JWT Some of the limitations of , Make sure you log out safely .

 

3. "One-stop" work style Redis challenges

redis Rich data structure , It generally does not cause functional problems . But as the number of requests increases ,SLA An increase in demand , We are bound to Redis Do some transformation and customization development .

3.1 High availability challenges

redis Provides master-slave 、 sentry 、cluster And so on , among cluster The model is the way most companies use now .

however ,redis Of cluster Pattern , There are a lot of hard injuries .redis cluster Adopt the concept of virtual slot , Put all the key Mapping to 0～16383 In an integer slot , It belongs to a decentralized architecture . But its maintenance cost is high ,slave Nor can it participate in read operations .

Its main problem , The limitation lies in some batch operations . because key By hash To multiple machines , therefore mget、hmset、sunion Such operations are very unfriendly , Performance problems often occur .

redis The master-slave mode is the simplest mode , But you can't do it automatically failover, Usually after master-slave switching , You also need to modify the business code , It's intolerable . Even with haproxy Such load balancing components , Complexity is also very high .

Sentinel mode when there are a large number of masters and slaves , Can significantly reflect its value . A cluster of sentinels , Able to monitor hundreds of clusters , However, it is difficult to maintain the sentinel cluster itself . Fortunately, ,redis The text protocol is very simple , stay netty in , Even directly provide redis Of codec. Self developed a sentinel system , Strengthen its function , It is feasible. .

3.2 Separation of hot and cold data

redis Is characterized by , Whatever the data , They all go into memory to do calculations , This is for the concept of time series , Business with hot and cold data , Caused a very large cost test . Why do most developers like to store data in MySQL in , instead of Redis in ？ In addition to transactional requirements , The big reason is the problem of historical data .

Usually , This switching of hot and cold data , It is completed by middleware . We also talked about ,Redis It's a text protocol , It's simple . Make a middleware , Or make a protocol compatible Redis Analog storage , It's easier .

For example, we Redis in , Keep only active users in the last year . An inactive user for several years , Suddenly accessed the system , When we get the data , You need middleware to transform , From larger capacity , Find in slower storage .

This is the time ,Redis The role of , It's more like a heat reservoir , More like a tradition cache What the layer does , When the business is already on scale . But notice , By this time , Our business layer code , It's always operational redis Of api. They use these numerous function instructions , I don't care whether the data is really stored in redis in , Still ssdb in .

3.3 Functional requirements

redis Can also play a lot of tricks . for instance , Full text search . Many people will choose es, but redis Ecology provides a module ：RediSearch, You can query , You can do it filter.

But we usually have more needs , Such as statistics 、 Search class 、 Operation effect analysis, etc . Such demands are related to big data , Even traditional DB Not competent . Now , Of course we have to redis Data in , Import to other platforms for calculation .

If you choose redis database , that dba Dealing with , Namely rdb, instead of binlog. There are a lot of rdb Parsing tool ( such as redis-rdb-tools), Be able to regularly rdb Parse into records , Import to hadoop And other platforms .

here ,rdb Become the hub of all teams , Become the basic data exchange format . Import to other db Business after , How to play, how to play , Not because the business system chooses redis It won't work .

 

4. summary

Most business systems , Run in the redis On , This is a lot that has been used MySQL Students who do business systems can't imagine . After reading the above introduction , I believe you can be right redis Have a general understanding of the storage functions that can be realized . Open your social app、 game app、 video app, Take a look at their functions , How much can it cover ？

What I want to emphasize here is , Some data , You don't have to land on RDBMS It's safe , They are not a strong demand .

Since the redis So powerful , Why do we have to mysql、tidb Such storage ？ The key lies in business attributes .

If a business system , Data for each interaction , Is a very large result set , And involves very complex Statistics 、 Filtering work , that RDBMS Is a must ; But if a system , Be able to pass through an identification , Quickly locate a class of data , This kind of data in the foreseeable future , It is limited. , That's perfect for Redis Storage .

An e-commerce system , choose redis Storage is death , But a social system is much happier . Choose the right tools in the right scene , That's what we should do .

But a system , Whether it can be in the product validation period , Can quickly respond to changes , Fast development online , Is the key to success . This is also used redis Do the database , The biggest benefit . Don't let the data loss scenario with very low probability , Scared . Compared to product success , Your system is as strong as steel , It's worthless .

Taste of little sister  (xjjdog), An official account that does not allow programmers to take turns. . Focus on infrastructure and Linux. Ten year structure , 10 billion traffic per day , Discuss the world of high concurrency with you , Give you a different taste . My personal wechat xjjdog0, Welcome to add friends , Further communication .