[the most comprehensive and detailed] how to design a database and table splitting scheme that can dynamically expand and shrink capacity?

For sub database and sub table , Mainly facing the following problems ：

• Choose a database middleware , research 、 Study 、 test ;• Design a plan for your sub database and sub table , How many banks do you want to divide , How many tables is each library divided into , such as 3 Databases , Each library 4 Tables ;• Based on the selected database middleware , And in the test environment to establish a good sub database sub table environment , Then test whether the reading and writing of sub database and sub table can be carried out normally ;• Complete the migration from single database and single table to sub database and sub table , Double writing scheme ;• Online system began to provide external services based on sub database and sub table ;• It's expanded , Expand to 6 Databases , Each library needs 12 Tables , How can you add more libraries and tables ？

This is one thing you have to face , That is, you have already worked out the plan of sub database and sub table , Then a pile of libraries and tables were built , Based on the sub database sub table middleware code development what are good , Test all ok 了 , Data can be evenly distributed to all databases and tables , And then you click on the system through a double written solution , Based on the sub database and sub table scheme .

So here's the problem , You can't support these libraries and watches anymore , What to do if we want to continue to expand capacity ？ This may mean that the capacity of each library is almost full , Or you have too much data , Maybe the write concurrency of each library is too high , You have to keep expanding .

This is all the things you have to go through to play on the sub database and sub table line .

analyse

Shut down and expand capacity （ Not recommended ）

This solution is the same as downtime migration , The steps are almost the same , The only thing is the derivative tool , It is to extract the data of the existing database tables and slowly pour it into the new database and tables . But it's better not to play like this , It's a bit unreliable , Because since Sub database and sub table That means the amount of data is too large , Maybe as many as a few hundred million , Even billions , You play like this , Something could go wrong .

When migrating from single database and single table to sub database and sub table , The amount of data is not very large , The biggest watch alone is twenty or thirty million . So you write a tool , Get more machines running in parallel ,1 Hours of data . There's no problem with that .

If 3 Databases + 12 Tables , Ran for a while , The amount of data 1~2 The hundred million . Light is the guide 2 100 million data , It's going to take a couple of hours ,6 spot , Just finished the data , We need to make subsequent configuration changes , Restart the system , Test verification ,10 It's time to finish . So we can't do this .

The optimized plan

Come up at the beginning 32 Databases , Each library 32 Tables , So the total is 1024 A watch .

I can tell you , This division , First of all , Basically, the Internet in China must be enough , second , Whether it's concurrent support or data volume support, there's no problem .

The normal write concurrency of each library is 1000, that 32 Each library can carry 32 _ 1000 = 32000 Write concurrent , If each library carries 1500 Write concurrent ,32 _ 1500 = 48000 Write concurrent , near 5 Write concurrency per second , Add one more MQ, Peak shaving , Write... Per second MQ 8 Ten thousand data , Consumption per second 5 Ten thousand data .

Some of these companies are not very top ranked in China , Their core system database , There may be hundreds of databases of this size ,128 Databases ,256 Databases ,512 Databases .

1024 A watch , Let's say that each table puts 500 All the data , stay MySQL You can put 50 Billion data .

Per second 5 Wan's writing concurrency , in total 50 Billion data , For most Internet companies in China , In fact, generally enough .

Talk about the expansion of sub database and sub table , For the first time, we divided the database and tables , Just give him enough at one time ,32 Databases ,1024 A watch , Maybe for most small and medium-sized Internet companies , It's been able to support for years .

One practice is to make use of  32 * 32  Let's divide the databases and tables , Divided into 32 Databases , Each table in the library is divided into 32 A watch . All together 1024 A watch . According to one id First, according to 32 Modulo routing to Library , According to 32 Modulo is routed to the table in the library .

orderId	id % 32 ( library )	id / 32 % 32 ( surface )
259	3	8
1189	5	5
352	0	11
4593	17	15

At the beginning , This library may be the logical library , Built on a database , It's just one. MySQL The server may have been built n Databases , such as 32 Databases . If you want to split it later , It's just being in storage and MySQL You can migrate between servers . Then the system can change the configuration .

For example, it can be extended to 32 Database servers , Each database server is a library . If it's still not enough ？ It can be extended up to 1024 Database servers , Each database server has a database and a table . Because at most 1024 Tables .

So make , You don't have to write your own code to do data migration , All to DBA Come and do it well , however DBA It is really necessary to do some work of database table migration , But it's better than writing your own code , Then the efficiency of data extraction is much higher .

Even if we want to reduce the number of warehouses , It's also very simple. , In fact, it's OK to shrink the volume by multiple , Then modify the routing rules .

Here is a summary of the steps ：

1. Set up several database servers , Several libraries on each server , How many tables per library , The recommendation is 32 library * 32 surface , For most companies , Maybe a few years .

2. Routing rules ,orderId model 32 = library ,orderId / 32 model 32 = surface

3. expansion , Apply for more database servers , packed MySQL, Expansion in multiples ,4 Servers , Expand to 8 Servers , Until then 16 Servers .

4. from DBA Be responsible for the database of the original database server , Migrate to a new database server , There are some convenient tools for library migration .

5. Let's change the configuration , Adjust the address of the database server where the migrated library is located .

6. Redistribute the system , go online , The original routing rules do not need to be changed , It can be directly based on n Multiple database server resources , Continue to provide services for online system .