Highly concurrent optimized Lua + openresty+redis +mysql (multi-level cache implementation) + current limit +canal synchronization solution

One 、 Multi level cache （lua + openresty+redis+mysql ）

Ideas ：
  1.  First from openresty The data in the cache of If you can get direct return .
  2.  If not openresty The data of , Again from redis Get data in  , If you can get , take redis Data stored in  openresty in , And then directly return to .
  3. If not redis Data in , Again from mysql In order to get , Store data to redis in , Back again .

1.1 lua Introduce

Lua Is a lightweight and compact scripting language , Using standard C Language written and open in source , It's designed to be embedded in the application , This provides the application with flexible extension and customization capabilities

1.2 openresty Introduce

It's based on nginx Of web platform , Mainly integrated lua Module It is directly provided to our developers . Can achieve ultra-high performance processing 1000K

OpenResty( also called ：ngx_openresty) It's based on nginx Scalable Web platform , Initiated by Chinese Zhang Yichun , Provides a lot of high quality third-party modules .

OpenResty Is a powerful Web application server ,Web Developers can use Lua Scripting language mobilization Nginx Various supported C as well as Lua modular , More importantly, in terms of performance ,OpenResty Sure Build up quickly enough to be competent 10K The ultra-high performance of the above concurrent connection response Web Application system .

360,UPYUN, Alibaba cloud , Sina , Tencent network , Where to go , Cool dog music, etc OpenResty Deep users .

OpenResty It is simply understood as It's equivalent to packaging nginx, And integrated LUA Script , Developers can implement related logic simply by providing modules , Not like before , Still need to be in nginx Write your own lua Script for , Call again .

Two 、 Current limiting （ Leaky bucket algorithm ）

In general , The concurrency of the home page is relatively large , Even with multi-level caching , When users keep refreshing the page , There is no need to , In addition, if a large number of malicious requests reach , It will also affect the system . And current limiting is one of the protective measures .

2.1 Comparison of current limit in life

• Dam discharge , Limit the flood flow through the gate （ Control the flow rate ）.

• To transact banking business ： Everyone gets the number first , Call processing at each window . The processing speed of each window depends on the specific business of the customer , Just wait in line for the call . If you are about to leave work , Tell the customer to come back tomorrow ( Reject traffic )

• The train station queued up to buy tickets for security check , By queuing up In turn .（ Cache processing task ）

2.2 nginx Current limiting of

nginx Two current limiting methods are provided ：

• One is to control the speed

• The second is to control the number of concurrent connections

2.3 One of the ways to control the rate ： Leaky bucket algorithm

(1) Leaky bucket algorithm realizes rate control and current limiting

Leaky bucket (Leaky Bucket) The algorithm is very simple , water ( request ) Enter the leaky bucket first , The leaky bucket comes out of the water at a certain speed ( The interface has a response rate ), When the water inflow speed is too high, it will overflow directly ( Access frequency exceeds interface response rate ), And then reject the request , It can be seen that the leaky bucket algorithm can forcibly limit the data transmission rate . The schematic diagram is as follows :

 

3、 ... and 、canal Synchronization solutions

3.1 Main uses ： be based on MySQL Database incremental log parsing , Provide incremental data subscription and consumption

   canal It can be used to monitor the change of database data , To get new data , Or modified data .canal It is due to Alibaba's dual computer room deployment in Hangzhou and the United States , There is a business requirement of cross machine room synchronization . Ali department company began to gradually try log parsing based on database , Get incremental changes to synchronize , This leads to incremental subscriptions & Consumer business .

3.2  principle ： according to mysql MS Architecture implementation simulation slave obtain master Of binlog Data analysis and processing

1. canal simulation mysql slave Interaction protocol , Pretend to be mysql slave, towards mysql master send out dump agreement
2. mysql master received dump request , Start pushing binary log to slave( That is to say canal)
3. canal analysis binary log object ( Originally byte flow )

 3.3 role （ Two ）

canal-server Server side Disguise himself as a malseter Of slave 
canal-client client Used to monitor canal-server client （java The client of ： Process data and business logic ）

3.4 canal Simple idea of synchronization ：

1. There's a database yes master 
2.canal-server yes slave( In disguise )
3.canal-client monitor canal-server 
4. Once the database master Data update occurs ,canal-server Get the data ,canal-client Monitor for data changes , Code implementation in the client ： Get data uniformly , Just synchronize

3.5 build canal And realize the steps of monitoring data changes ：

1. mysql Need to open binlog (master role )
2. mysql Create an account be used for salve Special use of , Grant authority slave Authority Remote authorization .
3. adopt docker install canal-server 
4. To configure canal-server( Configure connected to master Of ip And port And their own account and password, as well as the databases and tables to be monitored )
5. build canal-client(java Microservices ： monitor canal-server Get the modified data , Then do the business logic ： Sync count to redis in )
    1. Download the reference first , The installation starts from the local warehouse
    2. Add the start dependency Refresh
    3. Create a startup class to enable canalclient
    4. Create a listening class monitor canal-server Implement when something happens CRUD Execute business code when
    5. Configure the address and port of the server corresponding to the client
    

6.canal-client Write code to verify whether you can listen to the data .