当前位置:网站首页>Clickhouse synchronization MySQL (based on materialization engine)
Clickhouse synchronization MySQL (based on materialization engine)
2022-07-05 03:45:00 【Younger Cheng】
One 、Mysql engine ( Not recommended )
Synchronization considerations :1、 The library name specification cannot have “-”, The table name specification cannot have “-”
CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster]
ENGINE = MySQL('host:port', ['database' | database], 'user', 'password')
Parameter description :
host:port
— MySQL Service addressdatabase
— MySQL Database nameuser
— MySQL user namepassword
— MySQL User password
MySQL The engine will be remote MySQL Tables in the server / Library mapping to ClickHouse in ,MySQL The database engine will convert the query to MySQL Syntax and send to MySQL Server ( Equivalent to direct use of mysql)
Database synchronization :
CREATE DATABASE IF NOT EXISTS mysql_db ENGINE = MySQL ('localhost:3306', 'mysql', 'root', '123456');
Table synchronization :
CREATE TABLE IF NOT EXISTS tmp ENGINE = MergeTree ORDER BY id AS SELECT * FROM mysql('localhost:3306','test','user','root','123456')
Two 、MaterializedMySQL( To use , But the official is still in the experimental stage )
CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster]
ENGINE = MaterializedMySQL('host:port', ['database' | database], 'user', 'password') [SETTINGS ...]
[TABLE OVERRIDE table1 (...), TABLE OVERRIDE table2 (...)]
establish ClickHouse database , contain MySQL All the tables in , And all the data in these tables .
ClickHouse Server as MySQL Copy work . It reads binlog And implement DDL and DML Inquire about
1、mysql Turn on binlog and GTID
vim /etc/mysql/my.cnf
[mysqld]
# Appoint binlog Log storage location
#log-bin=/data/logs/mysql/mysql-bin.log
log-bin=/var/lib/mysql/mysql-bin
# Turn on GTID Pattern
gtid-mode=ON
# Set master-slave strong consistency
enforce-gtid-consistency=1
# Log
log-slave-updates=1
binlog_format=ROW
2、 Create a replication pipeline
# Start the materialization engine
SET allow_experimental_database_materialized_mysql=1;
CREATE DATABASE yfc
ENGINE = MaterializeMySQL('localhost:3306', 'yfc', 'root', '123456')
advantage : By monitoring mysql Of binlog file , Achieve incremental updates , Increased efficiency
Data restrictions :
1、 Sync mysql Before the data ,mysql Every watch of should have primary key( If there is no primary key , Error will be reported during synchronization )
2、MaterializedMySQL It is a library level engine , During synchronization, the table data in the whole database will be synchronized
3、mysql Data synchronization to clickhouse Index conversion will occur after : stay ClickHouse In the table ,MySQL Of PRIMARY KEY
and INDEX
Clause is converted to ORDER BY
Tuples
4、mysql In the transformation of clickhouse Table time , Each table will add two fields :_sign(1: write in 、-1: Delete ),_version
5、 stay clickhouse When adding synchronization in , There is no physical deletion , Only pass _sign Flag field to realize data filtering
6、 stay mysql conversion clickhouse when , Default ReplacingMergeTree engine , Ensure that no duplicate data appears
example :
#1、 Query the synchronized database
show databses;
use yfc;
show tables;
#2、 View the table creation statement
show create table sku_info;
#3、 Query synchronized data
select *,_sign,_version from sku_info;
#4、 The new data ( Go to mysql Of sku_info Insert data into the table )
INSERT INTO `yfc`.`sku_info`(`id`, `sku_code`) VALUES (3, '10003');
#5、 Modifying data
UPDATE `yfc`.`ts_store_info` set cust_name = ' Continuous commissioning test ' where id= 111;
#6、 Delete data
DELETE FROM `yfc`.`sku_info` where id= 3;
3、 ... and 、mysql Table function
Four 、datax
边栏推荐
- Dart series: collection of best practices
- [luat-air105] 4.1 file system FS
- SPI and IIC communication protocol
- Timing manager based on C #
- The perfect car for successful people: BMW X7! Superior performance, excellent comfort and safety
- Smart pointer shared_ PTR and weak_ Difference of PTR
- Technology sharing swift defense programming
- Subversive cognition: what does SRE do?
- Kubernetes -- cluster expansion principle
- About authentication services (front and back, login, registration and exit, permission management)
猜你喜欢
Operation flow of UE4 DMX and grandma2 onpc 3.1.2.5
Share the newly released web application development framework based on blazor Technology
程序员的视力怎么样? | 每日趣闻
【软件逆向-分析工具】反汇编和反编译工具
Redis source code analysis: redis cluster
51 independent key basic experiment
DMX parameter exploration of grandma2 onpc 3.1.2.5
Redis6-01nosql database
[C language] address book - dynamic and static implementation
[luat-air105] 4.1 file system FS
随机推荐
Unity implements the code of the attacked white flash (including shader)
线程基础知识
Redis6-01nosql database
JWT vulnerability recurrence
Flex flexible layout
Operation flow of UE4 DMX and grandma2 onpc 3.1.2.5
Use of kubesphere configuration set (configmap)
ICSI213/IECE213 Data Structures
[move pictures up, down, left and right through the keyboard in JS]
一文带你了解BI的前世今身与企业数字化转型的关系
v-if VS v-show 2.0
Nmap user manual learning records
About MySQL database connection exceptions
UE4 DMX和grandMA2 onPC 3.1.2.5的操作流程
[groovy] groovy environment setup (download groovy | install groovy | configure groovy environment variables)
[untitled]
Basic function learning 02
Web components series (VII) -- life cycle of custom components
Multi person online anonymous chat room / private chat room source code / support the creation of multiple chat rooms at the same time
[software reverse - basic knowledge] analysis method, assembly instruction architecture