I haven't seen this knowledge -- MySQL service evolution

MySQL Service evolution

1

summary

• When there's more and more data , And the growth rate is very fast , Especially when the concurrency is increasing , A single point of database will have performance bottlenecks , For example, there are not enough connections in the connection pool , Many requests are blocked , Queries that do not hit the index are time consuming , Physical storage also multiplies the pressure

• here , The database needs to be optimized , Hardware level , increase CPU、 Memory and storage space can solve certain problems in the early stage ;

• added , It needs to be optimized at the software level , Include ：

• SQL tuning , As mentioned earlier , Check out slow queries , Targeted optimization

• Table structure optimization , According to the business characteristics , Only reasonable data in the table is returned , Or the table is divided into multiple tables according to the business , in addition , Appropriate redundancy , It can also be reduced join, Improve query performance

• Read / write separation , For the actual business, read more and write less , Use binlog Synchronize multiple database instances from one master , Lifting performance

• Sub database and sub table , But when the data volume continues to increase , In particular, the single table data is very large , For example, over 500

2

Service evolution ...

• It's like a company growing up , The same is true of business volume , General , The normal evolution of a business system is as follows ：

• Single business system -- Single database ： here , Small business volume , The database has sufficient capacity and surplus

• Multi service system -- Single database ： This stage , Increased business complexity , Will split a complex business system into many sub business systems , Excellent database performance , It can continue to support multi business systems

• Multi service system -- Many database ： That is, by business 、 Business volume continues to grow , The database becomes a bottleneck , Divide the database into independent databases according to each business system , Independent deployment , Increase the service carrying capacity of each system , The carrying capacity of the overall business is also enhanced , This is the processing mode of microservices

• Sub database and sub table ： The business has entered the stage of sustained and rapid growth , The data volume of single or multiple business tables of some business subsystems exceeds the limit , Upgrade through hardware 、 Monomorphic SQL Optimize 、 Even when read-write separation , Can not effectively improve the performance of queries or operations , You need to use the split table ; There are mainly vertical sub table and horizontal sub table , Vertical table splitting is based on business characteristics , A single table only stores business-related fields ; When the amount of data is large enough , Exceed 500 Ten thousand records , The performance improvement of the vertical meter is limited , Finally, the horizontal sub table of sub database should be used

• Vertical sub table ： Split into multiple table structures by business , Make the table structure more business focused , Reduce the amount of data returned from queries

• Horizontal sub table ： The table structure is the same , Divide data based on different dimensions , The data is different ; It can also be divided into single database and multi database and multi table , Because a single library always has a performance bottleneck , Therefore, multi database and sub table are common

Business evolution process

1

Single application - single DB

• Common business forms of small projects

2

Multiple applications - single DB

• Enterprises have many common business forms of small projects , The database service needs relatively good performance

3

Single application - Master-slave DB

• Common patterns of medium-sized complex business systems , Read operations in the business 、 Write operations for different DB The server , Increase the capacity of business systems

• Lord 、 from DB The server should have a reasonable data synchronization mechanism

• You can switch directly in the business , You can also use middleware

4

Multiple applications - many DB

• From a business perspective, it will apply 、DB Cut them all , Slicing into smaller applications +DB Combine

• Microservice is a typical multi application - many DB The pattern of

5

Sub database and sub table

• When the application volume becomes larger and larger , The single table data of some businesses is very large , Exceed 1000 Ten thousand , At this point, it is necessary to consider the sub database and sub table

• Sub database and sub table bring great complexity to the application , Generally, you will use something like MyCat、ShardingSphere Wait for the middleware to handle , Shield the business DB Complexity

►►►

How to divide tables ？

• Realize the sub table , The core is to find a reasonable sub table key according to the business , Generally, several conditions should be considered ：

• Data can be evenly distributed in different databases or tables , And consider the characteristics of business growth

• Span query , Especially cross database join Query as few as possible

• The sub table key shall not change as much as possible

• Commonly used sub table key division ( Partition ) The policy has by scope 、hash、hash+ Scope, etc

Problems caused by sub table

• Cross database association query , If in a single warehouse , It can be used conveniently join Get related information 、 Generate required business reports , But after the table is divided, it may join The data is in another library ; Generally, appropriate redundant data is used 、 All libraries store global tables 、 Use the centralized data warehouse to process the summary business , And assembly in the business layer

• Distributed transactions , Transactions in a library are easy to handle , Cross Library 、 Cross business transactions are more complex ; Consider using message oriented middleware at the business level , And some third-party components Seata、TXLcn And so on

• Sort 、 Pagination 、 Sum up the problem of equal function calculation , Sub database calculation is required 、 Then summarize , Or use third-party components

• Distributed ID, After the sub-library sub-table , The original table primary key usage ( Such as the increase ) There may be problems , The main solutions are ：UUID、 Segment mode 、Redis cache 、 Snowflake algorithm (SnowFlake)

Third party plug-ins and middleware

• After the sub-library sub-table , It is actually very complicated to realize the data operation of multiple data sources in the business system

• There are third-party plug-ins or middleware in the industry to solve the problems caused by the above-mentioned database and table splitting

• ShardingSphere：Apache A project of open source , It needs to be embedded in the business system code for processing , It's based on Spring Boot Tools for , Intrusion into code

• MyCat： It is realized by setting up an intermediate layer between the business system and multi data sources with sub database and sub table , No code intrusion

---

  welfare