Mysql to PostgreSQL real-time data synchronization practice sharing

「 This is my participation 2022 For the first time, the third challenge is 17 God , Check out the activity details ：2022 For the first time, it's a challenge 」

Abstract ： quite a lot DBA And development students often encounter the problem of real-time synchronization from one database to another , Isomorphic data is also relatively easy , Encounter heterogeneous data 、 Watch more 、 When there is a large amount of data, it is difficult to synchronize . Recently learned about a real-time data synchronization tool Tapdata Cloud, Can be done very easily MySQL Real time data synchronization to PostgreSQL, Share with you , I hope it helps you .

This time MySQL Real time data synchronization to PostgreSQL It took only a few minutes to finish . The tool used is Tapdata Cloud , This tool is permanently free .

@[TOC](MySQL To PostgreSQL Real time data synchronization practice sharing )

First step ： To configure MySQL Connect

1. Click on Tapdata Cloud Operate the menu bar on the left side of the background 【 Connection management 】, Then click on the right area 【 Connection list 】 In the top right corner of the 【 Create connection 】 Button , Open the connection type selection page , And then choose MySQL

2. On the open connection information configuration page, enter the required configuration information in turn

【 even Pick up name call 】： Set the name of the connection , The names of multiple connections cannot be repeated

【 Database address 】： database IP / Host

【 End mouth 】： Database port

【 Database name 】：tapdata Database connection is based on db For a data source . there db Refers to... In a database instance database, Instead of a mysql example .

【 zhang Number 】： The account that can access the database

【 The secret code 】： The password corresponding to the database account

【 when between when District 】： The time zone of the database is used by default ; If time zone is specified , Use the specified time zone settings

Test connection , Prompt that the test passed

Save the connection after passing the test .

The second step ： To configure PostgreSQL Connect

1. The same as the first step , Click... On the left menu bar 【 Connection management 】, Then click on the right area 【 Connection list 】 In the top right corner of the 【 Create connection 】 Button , Open the connection type selection page , And then choose PostgreSQL

2. On the open connection information configuration page, enter the required configuration information in turn , After the configuration is completed, the test connection can be saved .

The third step ： Select synchronization mode - Total quantity / The incremental / whole + increase

Get into Tapdata Cloud Operate the background task management page , Click the add task button to enter the task setting process

According to the connection just built , Select the source side and target side .

According to data requirements , Select the library to be synchronized 、 surface , If you need to modify the table name , You can batch set the table names of the target end through the table name batch modification function in the page .

After the above options are set , Next, select the synchronization type , The platform provides full synchronization 、 The incremental synchronization 、 Total quantity + The incremental synchronization , Set the write mode and the number of reads .

If you choose the full amount + The incremental synchronization , After the full task is completed ,Tapdata Agent It will automatically enter the incremental synchronization state . In this state ,Tapdata Agent It will continuously monitor the data changes on the source side （ Include ： write in 、 to update 、 Delete ）, And write these data changes to the target in real time .

Click the task name to open the task details page , You can view task details .

Click task monitoring to open the task execution details page , You can view the task progress / Specific information about milestones, etc .

Step four ： Perform data check

Generally, after synchronization , I'm used to checking the data , Keep out of the puddles .

Tapdata Cloud There are three verification modes , I often use the fastest count check , Just select the table to be verified , There is no need to set other complex parameters and conditions , Easy and convenient .

If it's not enough , You can also select table full field value verification , In addition to selecting the table to be verified , You also need to set index fields for each table .

When verifying the full field value of the table , It also supports advanced verification . Through advanced verification, you can add JS Check logic , The source and target data can be verified .

There is also a verification method associated with field value verification , When creating associated field value verification , In addition to selecting the table to be verified , You also need to set index fields for each table .

I tested it myself MySQL Real time data synchronization to PgSQL Share your operations , I hope it helps you ！ It's not easy to code words , Reprint please indicate the source ~

Synchronization of other databases

Other database data can be synchronized to in real time Oracle、MySQL、PG、SQL Server、MongoDB、ES 、 Reach a dream 、Kafka、GP、MQ、ClickHouse、Hazelcast Cloud、ADB MySQL、ADB PostgreSQL、KunDB、TiDB、Dummy DB The way is to configure the connection between the source and the target first , Then create a new task and select the synchronization mode ： Total quantity / The incremental / Total quantity + The incremental , Because the steps are the same , The rest will not be explained . When you create a connection , Did you find ：DB2、Sybase、Gbase Several databases are now gray locked , It should be under development , The synchronization function of these databases may also be supported later .Tapdata Cloud Is a promise of permanent free use , But it's more demanding 、 Friends with a budget can also use the enterprise version , You can also apply for a trial . If you find it useful, please collect some praise and comments , If you have any database related questions, please feel free to contact me ~！