Ch single database data migration to read / write separation mode

1. Install two clickhouse, Two clickhouse The configuration is as follows （ Generally, you only need to install another one ）

stay /etc/clickhouse-server/config.xml In profile , stay remote_servers The node addition configuration is shown in the following figure ( Two clickhouse The examples are the same )

The following configuration is a shard Fragmentation , There are two in the first fragment replica copy

<remote_servers>
     <cluster1116>
         <shard>
            <internal_replication>true</internal_replication>
            <replica>
               <host>192.168.0.11</host>
               <port>9000</port>
               <user>default</user>
               <password>111111</password>
            </replica>
            <replica>
               <host>192.168.0.16</host>
               <port>9000</port>
               <user>default</user>
               <password>111111</password>
            </replica>
         </shard>
      </cluster1116>
</remote_servers>

To configure zookeeper( Two clickhouse The examples are the same )

<zookeeper>
      <node index="1">
         <host>192.168.0.11</host>
         <port>2181</port>
      </node>
</zookeeper>

Configuration Hong , On each instance , Fill in your own machine name

<macros>
  <layer>cluster86</layer>
  <shard>01</shard>
  <replica>aa-server11</replica>
</macros>

• layer Specify our cluster flag , Or use cluster keyword
• shard Specifies the configuration in the cluster fragmentation information , In the cluster, I configure 01
• replica Configure the backup synchronization node information of the current node

2. Create database

create database {new_database_name} ON CLUSTER {cluster_name};

In the above statement {new_database_name} Replace with the new database name ,{cluster_name} Replace with the cluster name

After creating the database through the above command , Both nodes of the cluster have this database （ The node that executes the command creates a new database and writes the command ZK in , Another node is from ZK To get information and create a database ）

The goal is

take

ATTACH TABLE t_0addc621
(
    `id` String,
    `reportName` String,
    `updateTime` Nullable(DateTime)
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(date)
ORDER BY id
SETTINGS index_granularity = 8192

Change to

CREATE TABLE IF NOT EXISTS aa_db.t_0addc621 on cluster chcluster
(
    `id` String,
    `reportName` String,
    `updateTime` Nullable(DateTime)
)
ENGINE = ReplicatedMergeTree('/clickhouse/aa_db/01/t_0addc621','{replica}')
PARTITION BY toYYYYMM(date)
ORDER BY id
SETTINGS index_granularity = 8192

Then execute the creation into the database

3. Migration table

The table engine used for the table creation statement of a single node cluster is MergeTree, The table engine with read-write separation should become ReplicatedMergeTree：

ENGINE = ReplicatedMergeTree('/clickhouse/aa_db/01/t_0addc621','{replica}')

• '/clickhouse/aa_db/01/t_0addc621' shard_name Segment name , The configuration of all nodes in the same partition of the cluster should be the same , Different watches must be different .

1. /clickhouse Default path
2. /aa_db Library name
3. /01 All machines in the same slice should remain the same (share, If you configure shard Parameters , It can also be used. {shard})
4. /t_0addc621 Table name

• '{replica}' replica_name Name of copy , Each node should be different , Generally, it is configured in the above macro replica Parameters , Use {replica} Just go

3.1 Get all the table creation statements in a single node and put them into a folder

It can be downloaded from ch Of metadata Folder /data/clickhouse/metadata obtain , You can also directly DBeaver Connect the single node library to get the table sql

3.2linux The server （ The previous single node one ） Create a new folder , And upload files

[[email protected] metadata]# mkdir -p /data/chtemp/metadata

[[email protected] metadata]# cd /data/chtemp/metadata

Upload the file to the folder

 

3.3 modify ch_table_change.sh, And implement , Build all tables sql The statement is modified to cluster 、 The engine is ReplicatedMergeTree Of

 [[email protected] metadata]# vi ch_table_change.sh

Execute... After saving

 [[email protected] metadata]# ./ch_table_change.sh

3.4 run sql Build table

 [[email protected] metadata]# for file in `ls | grep .sql`;do clickhouse-client --user default --password 1111111 -d aa_db --multiquery < $file;done

4. Migrating data ： modify ch_insert.sh, And implement

[[email protected] metadata]# vim ./ch_insert.sh

 

Execute... After saving

[[email protected] metadata]# ./ch_insert.sh

After the execution, the tables of the two nodes of the cluster will have data

Reference documents

ClickHouse ReplicatedMergeTree Family engine _ Seven years · The blog of -CSDN Blog _clickhouse replicatedmergetree

ClickHouse Replicated*MergeTree Copy table principle 、Distributed Principle of distributed table _ My name is Zhou Lidong's blog -CSDN Blog _clickhouse Copy table

ClickHouse And his friends （13）ReplicatedMergeTree Table engine and synchronization mechanism _ Lao Ye teahouse _ The blog of -CSDN Blog

ClickHouse actual combat -ReplicatedMergeTree Distributed table - Tencent cloud developer community - Tencent cloud

clickhouse-client Command line parameters and common cases _upupfeng The blog of -CSDN Blog _clickhouse-client