Jincang KFS data bidirectional synchronization scenario deployment

KFS Two way data synchronization scenario deployment

The two-way data synchronization scenario is that the databases at both ends are the source and target of each other ,A New incremental data on the end will be synchronized to B End ,B Incremental data on the end will also be synchronized to A End , And there is an anti loop mechanism before both ends , prevent A End incremental data is synchronized to B And then synchronize to A End , cause A End data duplication .KFS By establishing two one-to-one data links, which are the source end and the target end, the two-way data synchronization is realized , And pass the parameters property=replicator.log.slave.updates=false Prevent data loopback .

One 、 environmental information

1、 Software version

KFS edition :Kingbase FlySync V001R006C003B20220107

A End ：oracle 11.2.0.4.0

B End ：KingbaseES V008R006C003B0071

2、 Deployment Topology

KFS Centralized deployment with the database , Deploy on the same server .

3、IP Address

A End (oracle):192.168.100.111

B End (kes):192.168.100.114

Two 、A End (oracle) KFS Deploy

A End oracle As the source side redo Mode analysis .

( One )、 Environment configuration

1、 Create installation users

groupadd flysync

useradd flysync -g flysync -G mysql

passwd flysync

2、 Upload installation packages and license File to server /home/flysync Under the table of contents , And extract the

tar -xzvf KingbaseFlySync-V001R006C003B20220107-replicator.tar.gz

3、 Configuration will hosts file , Add both source and target servers

vi /etc/hosts

192.168.100.111 oracle

192.168.100.114 kes

4、 To configure /etc/security/limits.conf file 【 Optional 】

vi /etc/security/limits.conf

flysync - nofile 65535

flysync - nproc 8096

mssql - nofile 65535

mssql - nproc 8096

5、 Turn on the time synchronization service 【 Optional 】

yum install ntp

systemctl start ntpd

systemctl enable ntpd

6、 stay /etc/sysctl.conf To configure swappiness Parameters 【 Optional 】

vi /etc/sysctl.conf

vm.swappiness = 10

sysctl -p

7、 Check the dependent software jdk and ruby【 must 】

--jdk

java -version

Version less than 1.8 Of , have access to yum install 1.8 edition

yum install java-1.8.0-openjdk.x86_64

You can also download oracle Of 1.8 Version of JDK Manually upload and install packages

--ruby

have access to yum Way to install

yum install ruby

You can also use KFS The console server comes with ruby Package replacement , Position in ：/opt/KFS/console/media/rbenv,

Extract the package to /usr/local, Then configure /etc/profile In the document $PATH environment variable , take ruby Of bin Add directory to

vi /etc/profile

export PATH=$PATH:/usr/local/ruby/bin

When the configuration is complete , Use source Command application

source /etc/profile

（ Two ）、 Database configuration

1、 Configure database time format ：

ALTER SYSTEM SET NLS_DATE_FORMAT='YYYY-MM-DD' SCOPE=SPFILE;

2、 Create connected users and grant relevant permissions ：

Two way synchronization , The source side and the target side need to create a user respectively , To ensure that their intermediate tables do not conflict .

create user flysync identified by 123456;

alter user flysync default tablespace users;

alter user flysync quota unlimited on users;

GRANT CONNECT, RESOURCE TO FLYSYNC;

GRANT EXECUTE_CATALOG_ROLE TO FLYSYNC;

GRANT CREATE SESSION TO FLYSYNC;

GRANT SELECT ANY TRANSACTION TO FLYSYNC;

GRANT SELECT ANY TABLE TO FLYSYNC;

GRANT UNLIMITED TABLESPACE TO FLYSYNC;

GRANT DBA TO FLYSYNC;

GRANT CONNECT, RESOURCE TO FLYSYNC2;

GRANT EXECUTE_CATALOG_ROLE TO FLYSYNC2;

GRANT CREATE SESSION TO FLYSYNC2;

GRANT SELECT ANY TRANSACTION TO FLYSYNC2;

GRANT SELECT ANY TABLE TO FLYSYNC2;

GRANT UNLIMITED TABLESPACE TO FLYSYNC2;

GRANT DBA TO FLYSYNC2;

3、 Turn on Oracle Archive log of database （ Must be turned on ） 了 , The specific steps are as follows ：

(1). View the current archive mode

SQL>select  log_mode  from  v$database;

(2). Create archive Directory

[[email protected] oracle]$ mkdir -p $ORACLE_BASE/archivelog

[[email protected] oracle]$ sqlplus / as sysdba

SQL> alter system set log_archive_dest_1="location=/data/oracle/archivelog" scope=both;

(3). Restart the database to mount state

SQL>shutdown immediate

SQL>startup mount

(4). Change the database to archive mode

SQL>alter database archivelog;

(5). Open database

SQL>alter database open;

(6). Check whether the archive mode is on

SQL>select  log_mode  from  v$database;

4、 Open completion log ：

(1). Check the status of the completion log

SQL> SELECT supplemental_log_data_min, supplemental_log_data_pk,supplemental_log_data_all FROM v$database;

SUPPLEME SUP SUP

-------- --- ---

NO NO NO

(2). Open completion log ( It is suggested that MOUNT Execution in mode )

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;

SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

(3). Check the status of the completion log again

SQL> SELECT supplemental_log_data_min, supplemental_log_data_pk, supplemental_log_data_all FROM v$database;

SUPPLEME SUP SUP

-------- --- ---

YES      YES YES

must 3 Items are YES

5、 Switch log files

SQL> ALTER SYSTEM SWITCH LOGFILE;

（ 3、 ... and ）、KFS install

1、 To configure flysync.ini file

Configure... In the home directory of the installation user flysync.ini file

vi  /home/flysync/flysync.ini

Enter the following ：

[defaults]

user=flysync

install-directory=/home/flysync/kfsrep

profile-script=~/.bash_profile

rmi-port=11000

# Source side service

[oracle11g_source]

role=master

master=oracle

members=oracle

kufl-port=3112

replication-host=192.168.100.111

replication-port=1521

# The user name to connect to the database must be in upper case

replication-user=FLYSYNC

replication-password=123456

datasource-type=oracle

oracle-extractor-method=redo

datasource-oracle-service=orcl

svc-extractor-filters=dropstatementdata,ignoreddl

property=replicator.extractor.dbms.tablePatterns=FLYSYNC.*,TEST.*

property=replicator.extractor.dbms.keepMixDML=false

property=replicator.filter.ignoreddl.ignore=CREATE;ALTER;DROP;TURNCATE

property=replicator.extractor.dbms.lowLevelFilter=true

# Target side services

[oracle11g_target]

role=slave

master=kes

master-kufl-port=4112

members=oracle

kufl-port=4113

replication-host=192.168.100.111

replication-port=1521

replication-user=FLYSYNC2

replication-password=123456

datasource-type=oracle

datasource-oracle-service=orcl

svc-remote-filters=rename,casetransform

property=replicator.filter.casetransform.to_upper_case=true #oracle The table of is capitalized by default , So this parameter is configured as true

property = replicator.filter.rename.definitionsFile=/home/flysync/kfsrep/filters-config/oracle11g_target_rename.csv

property=replicator.log.slave.updates=false

2、 install

Switch to... Under the installation package directory tools subdirectories

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/tools

./fspm install

3、 take license Copy the file to the installation directory

cp license_8703_0.dat /home/flysync/kfsrep/license.dat

4、 modify rename file

Modify as B End to end services oracle11g_target Of rename file

vi /home/flysync/kfsrep/filters-config/oracle11g_target_rename.csv

flysync_kingbase8_source,trep_commit_seqno,*,FLYSYNC,-,-

flysync_kingbase8_source,consistency,*,FLYSYNC,-,-

flysync_kingbase8_source,heartbeat,*,FLYSYNC,-,-

flysync_kingbase8_source,trep_shard,*,FLYSYNC,-,-

flysync_kingbase8_source,trep_shard_channel,*,FLYSYNC,-,-

public,*,*,TEST,-,-

5、 start-up KFS

replicator start

see KFS state

replicator status

View service status

fsrepctl status

see KUFL list

kufl list

3、 ... and 、B End (KES v8r6)KFS Deploy

( One )、 Environment configuration

1、 Create installation users

groupadd flysync

useradd flysync -g flysync -G mysql

passwd flysync

2、 Upload installation packages and license File to server /home/flysync Under the table of contents , And extract the

tar -xzvf KingbaseFlySync-V001R006C003B20220107-replicator.tar.gz

3、 Configuration will hosts file , Add both source and target servers

vi /etc/hosts

192.168.100.111 oracle

192.168.100.114 kes

4、 To configure /etc/security/limits.conf file 【 Optional 】

vi /etc/security/limits.conf

flysync - nofile 65535

flysync - nproc 8096

mssql - nofile 65535

mssql - nproc 8096

5、 Turn on the time synchronization service 【 Optional 】

yum install ntp

systemctl start ntpd

systemctl enable ntpd

6、 stay /etc/sysctl.conf To configure swappiness Parameters 【 Optional 】

vi /etc/sysctl.conf

vm.swappiness = 10

sysctl -p

7、 Check the dependent software jdk and ruby【 must 】

--jdk

java -version

Version less than 1.8 Of , have access to yum install 1.8 edition

yum install java-1.8.0-openjdk.x86_64

You can also download oracle Of 1.8 Version of JDK Manually upload and install packages

--ruby

have access to yum Way to install

yum install ruby

You can also use KFS The console server comes with ruby Package replacement , Position in ：/opt/KFS/console/media/rbenv,

Extract the package to /usr/local, Then configure /etc/profile In the document $PATH environment variable , take ruby Of bin Add directory to

vi /etc/profile

export PATH=$PATH:/usr/local/ruby/bin

When the configuration is complete , Use source Command application

source /etc/profile

( Two )、 Database configuration

  Create a connection database account and authorize ：

ksql>CREATE USER FLYSYNC SUPERUSER PASSWORD '123456';

（ 3、 ... and ）、KFS install

1、 Check the database decoderbufs Is the plug-in installed

---- Check V8 Install under directory /lib Catalog , Is there a decoderbufs.so and kfs_current_query.so package , If there is no , take KingbaseFlySync Included with the installation package decoderbuf.so and kfs_current_query.so Copied to the Kingbase Installation directory /lib Under the table of contents .

---- Check V8 Install under directory /share/extension Catalog , Is there a decoderbufs.control file , If there is no , take KingbaseFlySync Included with the installation package decoderbuf.control Copied to the Kingbase Installation directory /share/extension Under the table of contents

ls   /opt/Kingbase/ES/V8/Server/lib/decoderbufs.so

ls   /opt/Kingbase/ES/V8/Server/lib/kfs_current_query.so

ls   /opt/Kingbase/ES/V8/Server/share/extension/decoderbufs.control

without , decompression KFS The compressed package included in the installation package , Copy these files to the appropriate directory

cd  /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/extensions/decoderbufs

cp kfs_decoderbufs_lib_linux64_v8r3.zip  /home/kingbase

cd /home/kingbase

unzip kfs_decoderbufs_lib_linux64_v8r3.zip

cp ./lib/decoderbufs.so /opt/Kingbase/ES/V8/Server/lib

cp ./lib/kfs_current_query.so /opt/Kingbase/ES/V8/Server/lib

cp ./share/extension/decoderbufs.control  /opt/Kingbase/ES/V8/Server/share/extension

chown kingbase:kingbase /opt/Kingbase/ES/V8/Server/lib/decoderbufs.so

chown kingbase:kingbase /opt/Kingbase/ES/V8/Server/lib/kfs_current_query.so

chown kingbase:kingbase /opt/Kingbase/ES/V8/Server/share/extension/decoderbufs.control

2、 Edit installation directory data/sys_hba.conf file , Add the following configuration , Enable the user to have copy permission :

vi /opt/Kingbase/data/sys_hba.conf

host replication all 127.0.0.1/32    md5

host replication all 0.0.0.0/0       md5

3、 Edit installation directory data/kingbase.conf file , Modify the following configuration ：

vi /opt/Kingbase/data/kingbase.conf

max_wal_senders=4          # Number of log sending processes , Number of databases *2, The minimum is 4

wal_keep_segments=4        # Number of logs retained , According to disk space settings , The bigger the better  

wal_level=logical          # The level of logging , It has to be for logical       

max_replication_slots=4    # Number of copy slots , Number of databases *2, The minimum is 4

Once the configuration is complete , Restart the database to make the configuration effective .

sys_ctl -D /opt/Kingbase/data restart

4、 To configure flysync.ini

Middle end flysync.ini The file configures two services , A target side service , Corresponding to the source end oracle11g; A source side service , Corresponding to the target end mysql8

[defaults]

install-directory=/home/flysync/kfsrep

profile-script=~/.bash_profile

rmi-port=11000

# Source side service

[kingbase8_source]

role=master

master=kes8

members=kes8

kufl-port=4112

replication-host=192.168.100.114

replication-port=54321

replication-user=FLYSYNC

replication-password=123456

datasource-type=kingbase

datasource-version=8

kingbase-extractor-method=xlogical

kingbase-dbname=TEST

svc-extractor-filters=replicate

property=replicator.filter.replicate.do=public.*,flysync_kingbase8.*

property=replicator.filter.replicate.ignore=public.T1

property=replicator.extractor.dbms.enableLob=false

# Large transaction splitting parameters , arrive 4000 Line start split transaction , The default is 500

property=replicator.extractor.dbms.minRowsPerBlock=4000

# Sync DDL Statement required , Supported can be configured DDL type

property=replicator.extractor.dbms.ddlListFile=/home/flysync/kfsrep/filters-config/ddl_support_list.csv

# Starting synchronization DDL Effective after ,create table as Type statements , At the same time produce DDL and DML, Whether to keep DML,true For the sake of reservation ,false In order not to keep , The default is true.

property=replicator.extractor.dbms.keepMixDML=true

# Target side services

[kingbase8_target]

role=slave

master=oracle

master-kufl-port=3112

members=kes8

kufl-port=3113

replication-host=192.168.100.114

replication-port=54321

replication-user=FLYSYNC

replication-password=123456

datasource-type=kingbase

datasource-version=8

kingbase-dbname=TEST

svc-parallelization-type=none

svc-remote-filters=casetransform,rename

property=replicator.filter.casetransform.to_upper_case=false

property = replicator.filter.rename.definitionsFile=/home/flysync/kfsrep/filters-config/kingbase8_target_rename.csv

property=replicator.log.slave.updates=false

property=replicator.applier.dbms.optimizeRowEvents=true

property=replicator.applier.dbms.maxRowBatchSize=5000

5、 from KFS Under the installation package directory extensions/jdbc Copy jdbc package

V8R3（ Not included ） The version above , Use kingbase8-8.6.0.jar

V8R3( contain ) The following versions , Use kingbase8-8.2.0.jar

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/extensions/jdbc

cp kingbase8-8.6.0.jar /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/flysync-replicator/lib

6、 install

cd /home/flysync/KingbaseFlySync-V001R006C003B20220107-replicator/tools

./fspm install

7、 Reply to license File to installation directory

cp license.dat /home/flysync/kesrep/license.dat

8、 modify rename file

modify B End as A End to end services kingbase8_target Of rename file

vi  /home/flysync/kfsrep/filters-config/kingbase8_target_rename.csv

FLYSYNC,trep_commit_seqno,*,flysync_kingbase8_target,-,-

FLYSYNC,consistency,*,flysync_kingbase8_target,-,-

FLYSYNC,heartbeat,*,flysync_kingbase8_target,-,-

FLYSYNC,trep_shard,*,flysync_kingbase8_target,-,-

FLYSYNC,trep_shard_channel,*,flysync_kingbase8_target,-,-

test,*,*,public,-,-

9、 Start and initialize

/home/flysync/kesrep/flysync/cluster-home/bin/startall

source ~/.bash_profile

see KFS state

replicator status

View service status

fsrepctl status

see KUFL list

kufl list

Four 、 Data relocation

Data from A To the end B End to end migration , This is done in two steps ： The first step is to migrate the table structure , The second step is to migrate data .

( One )、 Table structure migration

Table structure migration uses the extremely fast migration mode , stay B End execution .

ddlscan -target.service kingbase8_target -source.user FLYSYNC -source.pass 123456 -source.db orcl -source.dbtype oracle -source.host 192.168.100.111 -source.port 1521 -source.schema TEST  -target.db test -mgType 0

( Two )、 Stock data migration

Two way data synchronization scenario , Data migration can use extreme speed mode （ The target end moves rapidly ）, stay B End execution , such A The stock data of the terminal can be quickly obtained from A End migration to B End .

Speed mode relocation will lock the watch , Therefore, it is suggested to relocate during the period when the business is not busy .

B End execution ：

loader  -source.user FLYSYNC -source.pass  123456 -source.db TEST -source.dbtype oracle -source.host 192.168.100.111 -source.port 1521 -source.schema test -target.service kingbase8_target -mgType 0 -clean