Distributed | practice: smoothly migrate business from MYCAT to dble

author ： Xiao yaya

Love can be born DBA Team members , Be responsible for solving database failures and platform problems in the project , Have a special preference for database high availability and distributed technology .

In this paper, the source ： Original contribution

* Produced by aikesheng open source community , Original content is not allowed to be used without authorization , For reprint, please contact the editor and indicate the source .

Background introduction

There have been several problems in the customer environment recently , After discussion, it was decided to change the architecture , To put mycat Migrate to dble . The requirement is ： Minimum change .

Problem sorting

The following items to be considered are listed in combination with the customer's situation and requirements ：

• Parameter setting
• Piecewise functions
• The data of the data node
• Business SQL

Problem handling

1、 Parameter setting

mycat The end parameters are as follows :

<system>
<property name="defaultSqlParser">druidparser</property>
<property name="processors">4</property>
<property name="processorBufferPool">20480000</property>
<property name="processorBufferLocalPercent">100</property>
<property name="frontSocketSoRcvbuf">10485760</property>
<property name="frontSocketSoSndbuf">41943040</property>
<property name="frontSocketNoDelay">1</property>
<property name="backSocketSoRcvbuf">41943040</property>
<property name="backSocketSoSndbuf">10485760</property>
<property name="backSocketNoDelay">1</property>
<property name="maxPacketSize">2048576000</property>
<property name="memoryPageSize">100m</property>
</system>

Copy

For this parameter DBLE Side suggestions are as follows ：

• defaultSqlParser 、memoryPageSize 、processorBufferLocalPercent Has been in dble Abandoned in , No need to configure
• processorBufferPool The parameter name is changed to bufferPoolPageSize
• Other parameters are dble It can be found in server.xml China and mycat Maintain consistent configuration information , Details of some parameter information can be seen ：https://github.com/actiontech/dble-docs-cn/blob/2.19.11.0/tag/1.config_file/1.03_server.xml.md

Be careful ：mycat and dble The memory management configuration of is quite different . such as ： No, threadlocal Concept . It is recommended to read the following documents ：https://actiontech.github.io/dble-docs-cn/2.Function/2.07_memory_manager.html

2、 Piecewise functions

see mycat Fragmentation rule ：

<function name="mod-long" class="io.mycat.route.function.PartitionByMod">
<!-- how many data nodes -->
<property name="count">8</property>
</function>

Copy

Direct adoption DBLE Of hash Algorithm test found some data query errors ：

Check the data in the table ：

Sample data ：20210810143211157000000000036

Field ：user_code by varchar(32)

see mycat Environmental fragmentation rules ： The segmentation algorithm adopted is mod-long . and long The range of phi is zero -9223372036854774808~9223372036854774807 , Query error 20210810143211157000000000036 More than the long Value range of . So it can be concluded that mycat Special treatment has been made for this special situation .

In this case DBLE Side suggestions are as follows ：

1、 It is suggested to replace the fragmentation algorithm with HashString

The fragment column value exceeds Long The maximum of (9223372036854774807), It needs to be listed as String Type partition algorithm ;

Be careful ： Historical data needs to be re imported after changing algorithm segmentation

2、 use DBLE Custom algorithm

quote mycat Of PartitionByMod Algorithm , Join in dble Custom algorithm https://actiontech.github.io/dble-docs-cn/1.config_file/1.09_dble_route_function_spec.html?q= in .

The final configuration is as follows ：

<function name="test_func" class="com.actiontech.dble.custom.sharding.algorithm.PartitionByMod">
    <property name="count">8</property>
</function>

Copy

3、 Data node data

By introducing DBLE Custom split algorithm function , Compatible with the original environment mycat Split algorithm on , Therefore, the original mycat There is no need to make any changes to the storage node data of the environment . This greatly reduces the workload of our migration .

4、 Business SQL

After several rounds of testing , We found that mycat In many cases will SQL Directly distribute to the backend node , This caused us to encounter many problems in the test , because DBLE Subdivide various situations .

In order to ensure the accuracy of business query data to the greatest extent DBLE Cut a lot of these SQL Support for , But for the least change in business ,DBLE Also made appropriate opening .

Here are some records ：

• Yes insert into … select … Grammar support

at present DBLE The syntax support of vertical table is released .

• Yes rownum Transparent support

at present DBLE Release the specified table rownum Transparent .

• UPDATE query with sub-query is not supported

1.update Multi node update will trigger the problem of distributed transactions , It is difficult to guarantee consistency

2. According to the current implementation plan ,mycat about update+ The sub query uses the source sql Broadcast and distribute （ Distribute to all nodes associated with the table ）, This mechanism cannot guarantee the correctness in some cases

3. Multi table Association update is not supported ,dble As a middleware product , Dealing with a certain kind of sql, For this kind of sql The correct processing method is to distribute the sub query , Splice the outer layer after recycling the results sql To distribute , At present, this processing method is analyzed 、 It's hard to achieve , And performance issues also need to be considered .

4. In order to ensure update+ Sub queries such sql The correctness of the , It must not be used mycat This kind of broadcast is sent to do , So for this kind of sql At present, it is not supported

• Global table + Fragment table type SQL Support

ERROR 4004 (HY000): Unknown function FN_TREE_PATHNAME

1.dble stay 2.20.04.x Version and later versions have been optimized to directly implement this situation sql.

2. Global table + The partition table was not processed before because there may be problems with direct distribution , For global tables + Special scenes in the fragment table ： Global table + Vertical tables , It is not calculated that these nodes finally use the same node .（ Actually, it can be calculated ,dble stay 2.20.04.x Version and later have been optimized ）

3.mycat It will directly issue statements to nodes , The global table stores the same data on each configured node , If you split each node and table Left Join The results are simple UNION ALL Merge , It will cause data duplication , The accuracy of the data cannot be guaranteed . Therefore, it cannot be transmitted directly .

more

in the light of DBLE Yes MyCat More enhancements are recorded in ：https://actiontech.github.io/dble-docs-cn/0.overview/0.2_dble_enhance_MyCat.html