One 、 Push platform features
vivo The push platform is vivo The message push service provided by the company to developers , By establishing a stable network between the cloud and the client 、 Reliable long connection , Provide developers with services to push messages to client applications in real time , Support 10 billion level notifications / Message push , Second touch mobile users .
The push platform is characterized by high concurrency 、 Large amount of news 、 High timeliness of delivery . At present, the highest push speed 140w/s, Maximum daily message volume 150 Billion , End to end second online delivery rate 99.9%.
Two 、 Push platform Redis Introduction
be based on vivo Features of push platform , High requirements for concurrency and timeliness , And there are a lot of messages , Short message validity . therefore , The push platform chooses to use Redis Middleware is used as message storage and transfer , as well as token Information storage . Before, we mainly used two Redis colony , use Redis Cluster Cluster pattern . The two clusters are as follows :
Yes Redis The operation of , It mainly includes the following aspects :
1) Push link , Store the message body in the access layer to msg Redis colony , Message expiration time is msg Redis The expiration time of the stored message .
2) The push service layer goes through a series of logic , from msg Redis The cluster finds the message body , Inquire about client Redis colony client Information , If client On-line , Push... Directly . If client Not online , The message is id Write to the waiting queue .
3) If connected , Push service layer , Read waiting queue messages , Push .
4) Storage management services , It will be scanned regularly cii Indexes , according to cii Last update time stored , If 14 It hasn't been updated for days , Description is inactive user , Will clean up the token Information , At the same time, clean the token Corresponding waiting queue message .
Push operation Redis The flow chart is as follows :
3、 ... and 、 Push platform online problems
As described above , The push platform uses Redis The main msg Clusters and client colony , As the business grows , The system requires higher and higher performance ,Redis There are some bottlenecks , among msg Redis Cluster before optimization , The scale has reached 220 individual master,4400G Capacity . As the size of the cluster grows , Increased difficulty in maintenance , The accident rate is getting higher . especially 4 month , So and so star divorce , Real time concurrent messages 5.2 Billion ,msg Redis The number of single node connections in the cluster 、 Memory explosion problem , The number of connections of one node reaches 24674, Memory reaches 23.46G, continued 30 About minutes . period msg Redis The cluster response to read and write is slow , Mean response time 500ms about , Affect the stability and availability of the overall system , Availability reduced to 85%.
Four 、 Push platform Redis Optimize
Redis It is generally optimized from the following aspects :
1) Capacity :Redis It belongs to memory storage , Compared with disk storage database , Storage costs are expensive , It is precisely because of the memory storage feature that it has high read and write performance , But the storage space is limited . therefore , When the business is in use , It should be noted that the storage content should be thermal data as much as possible , And the capacity can be evaluated in advance , It's best to set the expiration time . When designing storage , Make rational use of the corresponding data structure , For some relatively large value, It can be compressed and stored .
2) heat key tilt :Redis-Cluster Map all the physical nodes to [0-16383]slot( Slot ) On , Each node is responsible for a part of slot. When there is a request to call , according to CRC16(key) mod 16384 Value , It was decided that key Request to which slot in . because Redis-cluster This feature , Each node is responsible for only part of slot, therefore , In the design key Make sure that key The randomness of , In particular, use some hash Algorithm mapping key when , Guarantee hash Random distribution of values . in addition , Control hotspots key Concurrency issues , Current limiting degradation or local caching can be adopted , Prevent hot spots key Concurrent requests are too high, resulting in Redis Hot spots tilt .
3) The cluster is too large :Redis-Cluster Use a no center structure , Each node holds the data and the entire cluster state , Each node is connected to all other nodes . Each node saves all nodes and slot The mapping relationship . When there are many nodes , Each node will also save more mapping relationships . The more data carried in the message body of the heartbeat packet between nodes . During expansion and contraction , The cluster is restarted clusterSlots It's a relatively long time . There is a risk of congestion in the cluster , Stability is affected . therefore , When using clusters , Try to avoid too many cluster nodes , Finally, the cluster is split according to the business .
Here's the problem : Why? Redis-Cluster Use 16384 individual slot, Not more , How many nodes can there be at most ?
The official author gives explain , And explain in the explanation ,Redis-Cluster No more than 1000 Main node .
Based on the above optimization directions , And its own business characteristics , The push platform starts from the following aspects Redis The road to optimization .
- msg Redis Cluster capacity optimization ;
- msg Redis Large clusters are split according to business attributes ;
- Redis hotspot key screening ;
- client Redis Cluster concurrent call optimization .
4.1 msg Redis Cluster capacity optimization
Mentioned above ,msg Redis The scale of the cluster reaches 220 individual master、4400G Capacity , The used capacity has reached... During the peak period 3650G, Used 83% about , If the subsequent push increases the amount , Capacity expansion is needed , The cost is too high . So for msg Redis Cluster storage content analysis , The analysis tool used is snowball RDB Analysis tools RDR .github website : Not much here , You can go github Download the corresponding tools from the website and use . This tool can analyze Redis Snapshot situation , Include :Redis Capacity of different structure types 、key Number 、top 100 largest keys、 Prefix key Quantity and capacity .
The conclusion after analysis :msg Redis In the cluster ,mi: The proportion of the structure at the beginning 80% about , Among them, the proportion of single push messages 80%. explain :
- Single push :1 Message push 1 Users
- Group push :1 A message can be repeatedly pushed to multiple users , Messages can be reused .
The feature of single push is one-to-one push , Push completed or failed ( Controlled 、 Invalid user, etc ) The message body is no longer used .
Optimization plan :
- Clean up the single push messages in time , If the user has received a single push message , received puback receipt , Delete directly Redis news . If a single push message is restricted from being sent for reasons such as control , Directly delete the single push message body .
- For messages with the same content , Aggregate storage , Store a message with the same content , news id It is used many times when pushing the logo .
After this optimization , The volume reduction effect is obvious . After the full volume goes online, the capacity is reduced 2090G, The original maximum capacity is 3650G, Reduced capacity 58%.
4.2 msg Redis Large clusters are split according to business attributes
Although the cluster capacity is optimized , But the rush hour msg Redis The pressure is still great .
Main cause :
1) Connect msg Redis There are many nodes , This leads to a high number of connections during peak periods .
2) The message body and the waiting queue are stored in a cluster , All push operations are required , Lead to Redis It's very concurrent , Fastigium cpu High load , arrive 90% above .
3) Old cluster Redis The version is 3.x, After break up , The new cluster uses 4.x edition . Compare with 3.x Version has the following advantages :
- PSYNC2.0: Optimized the previous version of , Switching between master and slave nodes will inevitably cause the problem of full replication .
- A new cache culling algorithm is provided :LFU(Least Frequently Used), The existing algorithms are optimized .
- Provides non blocking del and flushall/flushdb function , Effective solution deleted bigkey May cause Redis Blocking .
- Provides memory command , Achieve more comprehensive monitoring and statistics of memory .
- More memory saving , Store the same amount of data , Requires less memory space .
- Can do memory defragmentation , Gradually reclaim memory . When using Jemalloc Memory allocation scheme ,Redis You can use online memory defragmentation .
Splitting schemes are based on business attributes msg Redis Store information and split it , Split the message body and the waiting queue , Put them into two separate clusters . In this way, there are two splitting schemes .
Scheme 1 : Split the waiting queue from the old cluster
Just push the node to modify , But the sending waiting queue is continuous , A stateful , And clientId Online status related , Corresponding value Will be updated in real time , Switching will result in data loss .
Option two : Split the message body from the old cluster
All connections msg Redis Replace the node with a new address and restart , Push the node for double reading , Wait until the hit rate of the old cluster is 0 when , Directly switch to read new cluster . Because the message body is characterized by only write and read operations , No updates , Switching does not consider the state problem , As long as it can be written and read, there is no problem . And the message body capacity has the incremental attribute , Need to be able to expand capacity conveniently and quickly , The new cluster adopts 4.0 edition , Convenient for dynamic expansion and contraction .
Considering the impact on business and service availability , Make sure the message is not lost , Finally, we choose option 2 . The design adopts double reading and single writing scheme :
Because the message body is switched to the new cluster , That's during the switch for a while ( most 30 God ), The new message body is written to the new cluster , The old cluster stores the contents of the old message body . During this period, the push node needs double reading , Ensure that data is not lost . In order to ensure the efficiency of double reading , Need to support no code modification , Dynamic rule adjustment measures without restarting services .
The general rules are divided into 4 individual : Read only old 、 Read only new 、 Read the old before the new 、 Read the new before the old .
Design thinking : Server support 4 Strategies , Decide which rule to follow through the configuration of the configuration center .
The judgment basis of the rule : It is determined according to the hit number and hit rate of the old cluster . Initial online rule configuration “ Read the old before the new ”; When the hit rate of the old cluster is lower than 50%, Switch to " Read the new before the old "; When the number of hits in the old cluster is 0 after , Switch to “ Read only new ”.
The hit rate and hit number of old clusters are increased through general monitoring .
The flow chart of scheme 2 is as follows :
Split effect :
- Before splitting , The old msg Redis The peak load of the cluster in the same period 95% above .
- After break up , At the same time, the peak load is reduced to 70%, falling 15%.
Before splitting ,msg Redis The average response time of the cluster in the peak period in the same period 1.2ms, There are calls during peak hours Redis Slow response . After break up , The average response time is reduced to 0.5ms, There is no problem of slow response during peak hours .
4.3 Redis hotspot key screening
I said before ,4 Star hot events in August , appear msg Redis Number of single node connections 、 Memory explosion problem , The number of single node connections has reached 24674, Memory reaches 23.46G.
because Redis Virtual machines used by the cluster , At first, it was suspected that the host of the virtual machine was under pressure , Because according to the troubleshooting, the node with the problem is mounted on the host Redis There are many master nodes , Probably 10 about , While other host computers mount 2-4 Left and right master nodes , So for master A round of equalization optimization is carried out , Make the master nodes allocated by each host more balanced . After equalization , The overall situation has improved to a certain extent . however , At push peak , Especially when pushing at full speed and full volume , There are still occasional single node connections 、 Memory explosion problem . Observe the incoming and outgoing traffic of host network card , There are no bottlenecks , At the same time, it also eliminates the influence of other business nodes on the host . Therefore, it is doubtful that business use Redis There is a hot spot tilt problem .
Monitor the call chain by crawling during peak hours , You can see it in the picture below , We 11:49 To 12:59 During this period, call msg Redis Of hexists Commands are time-consuming , This command is mainly used to query whether the message is in mii Index , Link analysis is time-consuming key Mostly mii:0. At the same time, the problem nodes Redis Memory snapshot for analysis , Find out mii:0 The proportion of capacity is very high , There is a read mii:0 Hotspot issues .
After analysis and investigation , Discovery generates messages id Generated by snowflake algorithm messageId, There is a tilt problem , Because the sequence values of the same millisecond are from 0 Start , And the sequence length is 12 position , Therefore, for the management background and..., the concurrency is not very high api node , Generated messageId Basically the last 12 Position as 0. because mii Indexes key yes mi:${messageId%1024},messageId Last 12 Position as 0,messageId%1024 That is to say 0, This leads to msg Redis in mii:0 This key It's big , High hit rate during query , So it led to Redis Heat of key problem .
Optimization measures :
1) Snowflake algorithm transformation , Generate messages id The use of sequence The initial value is no longer 0, But from 0~1023 Take a random number , Prevent hot spot tilt problems .
2) adopt msg The message type in the message body and the existence of the message body are used to replace the call hexists command .
Final effect : After optimization ,mii The index is evenly distributed ,Redis The number of connections is stable , Memory growth is also relatively stable , No more Redis Single node memory 、 Connection explosion problem .
4.4 client Redis Cluster concurrent call optimization
The upstream node calls the push node through clientId Be consistent hash Called , The push node caches clientInfo Information to local , Cache time 7 God , When pushing , Query local cache first , Judge that client Whether it works . For important and frequently changing information , Direct inquiry client Redis obtain , This leads to push peak ,client Redis There's a lot of pressure on the cluster , High concurrency ,cpu High load .
Before optimization, push node operation cache and client Redis flow chart :
Optimization plan : To the original clientInfo Split the cache , Split into three caches , Adopt a graded scheme .
- cache Or save the original clientInfo Some of the information , The information changes infrequently , Cache time is still 7 God .
- cache1 cache clientInfo Constantly changing information , Such as : Online status 、cn Address, etc .
- cache2 cache ci Encrypt some parameters , This part of the cache is only used when encryption is required , The frequency of change is not that high , Changes only when connected .
Due to the new cache , Cache consistency needs to be considered , So add a new measure :
1) Push cache verification , call broker node , according to broker The return information of , Update and clean up local cache information .broker New offline 、aes Mismatch error code . Next push or retry , Will come back from Redis Load in , Get the latest client Information .
2) According to the uplink event of the mobile terminal ,connect and disconnect when , Update and clean up local cache information , Next push or retry , Will come back from Redis Load in , Get the latest client Information .
Overall process : When the message is pushed , Query local cache first , Cache does not exist or has expired , Only from the client Redis Load in . Pushed to the broker when , according to broker Return information , Update or invalidate the cache . The upside , received disconnect、connect event , Update or invalidate the cache in time , Push again from client Redis load .
After optimization, push node operation cache and client Redis flow chart :
Optimized effect :
1) newly added cache1 cache hit rate 52%,cache2 cache hit rate 30%.
2)client Redis The amount of concurrent calls has been reduced by nearly 20%.
3) Fastigium Redis Load reduction 15% about .
5、 ... and 、 summary
Redis Because of its high concurrency performance and supporting rich data structures , In high concurrency system, as cache middleware, it is a better choice . Of course ,Redis Whether it can play a high performance , It also depends on whether the business really understands and uses Redis. There are several points to pay attention to :
1) because Redis Cluster pattern , Each master node is responsible for only part of slot, Business in design Redis key Give full consideration to key The randomness of , Evenly dispersed in Redis On each node , At the same time, avoid large key appear . in addition , Business should avoid Redis Request hot issues , Request to call a few nodes at the same time .
2)Redis The actual throughput is also related to the request Redis Packet data size , Network card related , Official documents There are relevant instructions , The size of a single package exceeds 1000bytes when , Performance can degrade dramatically . So it's using Redis Try to avoid big key. in addition , It's best to base on the actual business scenario and the actual network environment , Pressure test the performance of bandwidth and network card , Find out the actual throughput of the cluster .
With our client Redis Clusters, for example :( For reference only )
- Network:10000Mb;
- Redis Version:3.x;
- Payload size:250bytes avg;
- command :hset(25%)、hmset(10%)、hget(60%)、hmget(5%);
- Performance : The number of connections 5500、48000/s、cpu 95% about .
Redis There is less support in real-time analysis , In addition to basic indicator monitoring , Real time memory data analysis is temporarily not supported . In the actual business scenario, if Redis bottleneck , Often the monitoring data is also missing , Positioning problems are difficult . Yes Redis Our data analysis can only rely on analysis tools for Redis Snapshot files for analysis . therefore , Yes Redis The use of depends on the business to Redis A full understanding of , When designing the scheme, fully consider . At the same time, according to the business scenario Redis Conduct performance pressure test , Know where the bottleneck is , Prepare for monitoring and capacity expansion and contraction .
author :vivo Internet server team -Yu Quan