High concurrency architecture design

1、 vertical direction ： Improve single machine capability

Improving the processing capacity of a single machine can be divided into two aspects: hardware and software ：

Hardware direction , Upgrade the server hardware , Buy multi-core high-frequency machines , Large memory , Large capacity disk, etc .

Software direction , Including faster data structures （ Concurrent programming at the programming language level ）, Improve the architecture , Application multithreading 、 coroutines （select/poll/epoll etc. IO Multiplexing technology ）, And on the performance optimization of various means , But this approach is prone to bottlenecks .

2、 horizontal direction ： Distributed cluster

In order to solve the complexity problem of distributed system , Architecture layering and service splitting are commonly used , Isolate by layering , Decouple through microservices .

There is no upper limit in theory , As long as we do a good job in the division of levels and services , With the expansion of the machine, we can meet the demand , But it's not , On the one hand, distributed will increase the complexity of the system , On the other hand, after the scale of the cluster goes up , It will also introduce a bunch of service discovery 、 New issues in service governance .

Because of the vertical limitation , therefore , We usually focus more on horizontal expansion , The implementation of high concurrency system mainly focuses on the horizontal direction .

04 clustering
The hardware expansion cost of a single machine is high , Software optimization is prone to performance bottlenecks , Therefore, cluster is used to solve the problem of high concurrency . Load balancing is a common solution , That is, the front-end traffic is allocated to different service nodes .

Under the cluster structure , Pooling can be used （ Memory pool , Connection pool , Thread pool ）, Distributed cache , Distributed message queuing , Flow control technology （ service degradation , Applied resolution , Current limiting ） High concurrency with database （ Sub database and sub table , Separation of reading and writing, etc ） Improve concurrency .

Load balancing can be divided into 3 Kind of ：

1、DNS Load balancing , Client pass URL When a network service request is initiated , Will go to DNS The server does domain name interpretation ,DNS Will follow a certain strategy （ For example, proximity strategy ） hold URL convert to IP Address , The same URL Will be interpreted as different IP Address , This is DNS Load balancing , It's a coarse-grained load balancing , It only uses URL First half part , because DNS Load balancing generally adopts the principle of proximity , So the delay is usually reduced , but DNS Yes cache, So it will also update the problems that are not timely .

2、 Hardware load balancing , By arrangement F5,A10 And other special load balancing equipment to the machine room for load balancing , High performance , But it's expensive .

3、 Software load balancing , Using software to realize four layer load balancing （LVS） And seven layers of load balancing （Nginx）.

05 Pool technology
5.1 Memory pool
How to create a memory pool ：

1、 Use memory mapping for large blocks of memory requested by the user

2、 For small pieces of memory, take them from the appropriate linked list in the memory pool

notes ：Linux It has its own memory management mode , But the system level memory optimization technology is far from meeting the actual needs , Popular memory optimization techniques include tcmalloc、ptmalloc、jemalloc etc. .

The role of memory pools ：

1、 Store large pieces of data

2、 Store data cache

5.2 process / Thread pool
The role of process pool and thread pool ：

1、 Avoid the time overhead of dynamic startup

2、 Make the processing more simple

3、 Make full use of hardware resources

Precautions for process pool and thread pool ：

1、 Typical producer consumer problems

2、 Note the competition for access to shared resources

5.3 Connection pool
Connection pool is the technology of creating and managing a connection buffer pool , These connections are ready to be used by any thread that needs them , For example, database connection pool .

How to create a connection pool ：

1、 Preassigned connections for fixed data

2、 Allocate corresponding resources to each connection

The role of connection pooling ：

1、 Speed up the creation of new connections

2、 It can be used for permanent connections within the cluster

06 cache
Cache can be divided into local cache and distributed cache .

Local cache ： Programming to realize （ Member variables 、 local variable 、 Static variables ）.

Distributed cache ： With the help of Redis、Memcache Realization .

In general, the write request is far less than the read request , For the scenario of writing less and reading more , It is very suitable to introduce cache cluster . Write a copy of data to the cache cluster while writing the database , Then use the cache to carry most of the read requests , When the cache does not exist, go to the database to find , This is done by caching the cluster , You can use fewer machine resources to carry higher concurrency .

The hit rate of cache is generally very high , And very fast , It's also very good at processing （ It's easy to do tens of thousands of concurrent on a single machine ）, It's the ideal solution .

Of course , When using distributed caching , Special attention needs to be paid to dealing with consistency issues , Cache breakdown , Cache penetration , Cache avalanches and so on .

07 Message queue
7.1 summary
Distributed cache has excellent performance in scenarios with more reads and less writes , For scenarios with many write operations, message queue clusters can be used , It can handle write request asynchronously , To achieve the effect of peak cutting and valley filling .

Message queuing can do decoupling , In a scenario where only final consistency is required , It's very suitable for flow control .

There are many famous message middleware in the industry , such as ZeroMQ,rabbitMQ,kafka etc. .

7.2 characteristic
1、 Business coupling ;

2、 Final consistency ;

3、 radio broadcast ;

4、 Peak shifting and flow control .

08 Flow control
When the service appears

8.1 service degradation
Auto downgrade ： Overtime 、 Number of failures 、 fault 、 Current limiting

Manual degradation ： seckill 、 double 11 Big promoting etc.

Problems to be considered in service degradation ：

1、 Core services 、 Non core services

2、 Whether downgrade is supported , Downgrade strategy

8.2 Applied resolution
Apply the split principle ：

1、 Business first ;

2、 Step by step ;

3、 Give consideration to technology ： restructure 、 layered ;

4、 Reliability test

8.3 Current limiting
Common treatment methods for current limiting are ： Counter 、 The sliding window 、 Leaky bucket 、 token .

1、 Counter method

Counter is a relatively simple current limiting algorithm , Over a period of time , Count , Compare with the threshold , At the critical point of time , Clear the counter 0.

however , There is a time critical point in the counter method . such as , stay 11:50:00 To 11:59:59 There are no user requests during this time , And then in 12:00:01 This instant sends out 1000 A request ,12:00:59 Again 1000 A request , At this critical point, a large number of requests from malicious users may be tolerated , Even more than the system expected .

2、 The sliding window

Because the counter has a critical point defect , Later, the sliding window algorithm appeared to solve .

The sliding window

Sliding window means to put a fixed time slice , division , And as time goes by , For mobile , This cleverly avoids the critical point of the counter . That is, these fixed number of movable grids , Will count to determine the threshold , Therefore, the number of grids affects the accuracy of sliding window algorithm .

3、 Leaky bucket algorithm

Although the sliding window effectively avoids the problem of time critical point , But there is still the concept of time slice , Leaky bucket algorithm is better than sliding window in this respect , More advanced .

Leaky bucket algorithm

There is a fixed barrel , The rate of inflow is uncertain , But the rate of outflow is constant , When the water is full, it will overflow .

4、 Token bucket algorithm

In a sense , Token bucket algorithm is an improvement of leaky bucket algorithm , The bucket algorithm can limit the rate of request calls , The token bucket algorithm can limit the average rate of calls and also allow a certain degree of burst calls .

In token bucket algorithm , There is a bucket , Used to store a fixed number of tokens . There is a mechanism in the algorithm , Put token into bucket at a certain rate . Each request call needs to get the token first , Only get the token , Only then has the opportunity to carry on , Otherwise, choose to wait for the available token 、 Or just refuse .

09 Database high concurrency
Database high concurrency is divided into single machine high concurrency （ Mainly the storage engine implementation ） High concurrency with cluster ：

1、 Single machine high concurrency

InnoDB The storage engine adopts multi version concurrency control technology （MVCC） Without lock , Realize concurrent reading and writing , At the same time, the concurrency efficiency is controlled through the transaction isolation level .

2、 Cluster high concurrency

The high concurrency of database clusters is mainly achieved by dividing databases and tables 、 Active / standby read / write separation .