当前位置:网站首页>Clickhouse同步mysql(基于物化引擎)
Clickhouse同步mysql(基于物化引擎)
2022-07-05 03:33:00 【Younger成】
一、Mysql引擎(不推荐)
同步注意事项:1、库名规范不能有“-”,表名规范不能有“-”
CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster]
ENGINE = MySQL('host:port', ['database' | database], 'user', 'password')
参数说明:
host:port
— MySQL服务地址database
— MySQL数据库名称user
— MySQL用户名password
— MySQL用户密码
MySQL引擎将远程的MySQL服务器中的表/库映射到ClickHouse中,MySQL数据库引擎会将对其的查询转换为MySQL语法并发送到MySQL服务器中(等同于直接使用mysql)
数据库同步:
CREATE DATABASE IF NOT EXISTS mysql_db ENGINE = MySQL ('localhost:3306', 'mysql', 'root', '123456');
表同步:
CREATE TABLE IF NOT EXISTS tmp ENGINE = MergeTree ORDER BY id AS SELECT * FROM mysql('localhost:3306','test','user','root','123456')
二、MaterializedMySQL(好用,但是官方还在实验阶段)
CREATE DATABASE [IF NOT EXISTS] db_name [ON CLUSTER cluster]
ENGINE = MaterializedMySQL('host:port', ['database' | database], 'user', 'password') [SETTINGS ...]
[TABLE OVERRIDE table1 (...), TABLE OVERRIDE table2 (...)]
创建ClickHouse数据库,包含MySQL中所有的表,以及这些表中的所有数据。
ClickHouse服务器作为MySQL副本工作。它读取binlog并执行DDL和DML查询
1、mysql开启binlog和GTID
vim /etc/mysql/my.cnf
[mysqld]
# 指定binlog日志存储位置
#log-bin=/data/logs/mysql/mysql-bin.log
log-bin=/var/lib/mysql/mysql-bin
# 开启GTID模式
gtid-mode=ON
# 设置主从强一致性
enforce-gtid-consistency=1
# 记录日志
log-slave-updates=1
binlog_format=ROW
2、创建复制管道
#开启物化引擎
SET allow_experimental_database_materialized_mysql=1;
CREATE DATABASE yfc
ENGINE = MaterializeMySQL('localhost:3306', 'yfc', 'root', '123456')
优点:通过监听mysql的binlog文件,实现增量更新,提升了效率
数据限制:
1、同步mysql数据之前,mysql的每张表应该都有primary key(如果没有主键,同步会报错)
2、MaterializedMySQL是库级别的引擎,同步的时候会以全库中表数据同步
3、mysql数据同步到clickhouse后会发生索引转换:在ClickHouse表中,MySQL的 PRIMARY KEY
和 INDEX
子句被转换为 ORDER BY
元组
4、mysql在转换为clickhouse表的时候,每张表都会新增两个字段:_sign(1:写入、-1:删除),_version
5、在clickhouse中同步新增时,并没有实现物理意义上的删除,只通过_sign标志字段来实现数据过滤
6、在mysql转化clickhouse时,默认使用的ReplacingMergeTree引擎,保证没有重复数据出现
实例:
#1、查询同步的数据库
show databses;
use yfc;
show tables;
#2、查看建表语句
show create table sku_info;
#3、查询同步的数据
select *,_sign,_version from sku_info;
#4、新增数据(往mysql的sku_info表中插入数据)
INSERT INTO `yfc`.`sku_info`(`id`, `sku_code`) VALUES (3, '10003');
#5、修改数据
UPDATE `yfc`.`ts_store_info` set cust_name = '连调测试' where id= 111;
#6、删除数据
DELETE FROM `yfc`.`sku_info` where id= 3;
三、mysql表函数
四、datax
边栏推荐
- [groovy] groovy environment setup (download groovy | install groovy | configure groovy environment variables)
- Delphi read / write JSON format
- Sqoop命令
- [groovy] string (string type variable definition | character type variable definition)
- How to make the listbox scroll automatically when adding a new item- How can I have a ListBox auto-scroll when a new item is added?
- Basic knowledge of tuples
- Talk about the SQL server version of DTM sub transaction barrier function
- Difference between MotionEvent. getRawX and MotionEvent. getX
- Class inheritance in C #
- Jd.com 2: how to prevent oversold in the deduction process of commodity inventory?
猜你喜欢
Sqoop命令
Azkaban安装部署
Class inheritance in C #
SQL performance optimization skills
Accuracy problem and solution of BigDecimal
el-select,el-option下拉选择框
Watch the online press conference of tdengine community heroes and listen to TD hero talk about the legend of developers
v-if VS v-show 2.0
Linux安装Redis
De debugging (set the main thread as hidden debugging to destroy the debugging Channel & debugger detection)
随机推荐
How to define a unified response object gracefully
【web审计-源码泄露】获取源码方法,利用工具
Azkaban实战
Azkaban安装部署
FBO and RBO disappeared in webgpu
The perfect car for successful people: BMW X7! Superior performance, excellent comfort and safety
Talk about the SQL server version of DTM sub transaction barrier function
Azkaban概述
Clean up PHP session files
[learning notes] month end operation -gr/ir reorganization
Simple use of devtools
51 independent key basic experiment
Sqoop command
The latest blind box mall, which has been repaired very popular these days, has complete open source operation source code
2. Common request methods
Multimedia query
El tree whether leaf node or not, the drop-down button is permanent
040. (2.9) relieved
Devtools的簡單使用
问下,这个ADB mysql支持sqlserver吗?