Kunlunbase instruction manual (III) data import & synchronization

One 、 Environmental Science

From the original PostgreSQL Database migration business data to KunlunBase colony . because KunlunBase compatible postgreSQL agreement , Therefore, it can directly support Pg Data import and export tool ：pg_dump and pg_restore.

But because of KunlunBase Currently compatible PostgreSQL The version is 11.5, So for inconsistent versions ： Newer versions of tools can be compatible with older versions , Older versions of tools are not supported .

For online real-time data synchronization ,KunlunBase Support Oracle GoldenGate, have access to Oracle GoldenGate Take data from PostgreSQL Or some other database （Oracle/SQL Server/MySQL etc. ） Synchronize to KunlunBase.

Two 、 Data export and import steps

because KunlunBase When inserting data, the table is required to have a primary key , To ensure the success of data migration , Required in native PostgrSQL The process of exporting data from the database is divided into two steps （ First step , Export data structure ,  The second step , Export actual data ）.

During the recovery process ： Restore the data structure first , Importing data , Next, the specific operation instructions will be demonstrated in two scenarios .

2.1  In tables

2.1.1  Derived table structure

pg_dump -h 192.168.0.142 -p 5432 -tcustomer1_3 -t customer1_4 postgres-s>customer1_34str.sql （-s  The parameter is only inverted table structure ）


2.1.2  Export table data

pg_dump -h 192.168.0.142 -p 5432 -tcustomer1_3 -t customer1_4 postgres -a >customer1_34_data.sq


2.1.3  Restore table structure

psql --dbname=bktest3 --host=192.168.0.142 --port=5401 -U abc -f customer1_34str.sql


2.1.4  Restore table data

psql --dbname=bktest3 --host=192.168.0.142 --port=5401 -U abc -f customer1_34_data.sql


2.2  In terms of databases

2.2.1  Export full table structure

pg_dump -d postgres -U postgres -s>postgres_str.sql


2.2.2  Export the table data of the whole library

pg_dump -d postgres -U postgres -a>postgres_data.sql


It's fine too dump And tar Format export , Reduce space and improve performance ：

pg_dump -d postgres -U postgres -a -Fc >postgres_data.dump ( Minimum size )
pg_dump -d postgres -U postgres -a -Ft >postgres_data.tar


2.2.3  Restore the whole database table structure

psql --dbname=bktest2 --host=192.168.0.142 --port=5401 -U abc -f postgres_str.sql


2.2.4  Restore all database table data

psql --dbname=bktest2 --host=192.168.0.142 --port=5401 -U abc -f postgres_data.sql


If it is dump Format exported data , Need to use pg_restore:

pg_restore -h 192.168.0.142 -p 5401 -U abc-d bktest2 postgres_data.dump


(pg_restore Only downward compatible , The high version of the pg_dump The exported file cannot use a lower version pg_restore recovery ）

2.3 Oracle GoldenGate Real time synchronization PostgreSQL To KunlunBase

2.3.1  Installation and deployment Oracle GoldenGate

Need to download and install OGG for PostgreSql Microservice version ,oggs_Linux_x64_PostgreSQL_services_shiphome edition . Install according to the official installation manual .

The installation and deployment architecture is shown in the figure below ：

2.3.2  To configure PostgreSQL Data sources and KunlunBase The goal is

Source database information ：ip Address  192.168.0.10, Copy users ：postgres Database name ：PGSOUR

Target database information ：ip Address  192.168.0.12, Copy users ：postgres Database name ：Kunlundb

Set database parameters ：

alter system set wal_level='logical';
alter system set max_replication_slots = 20;


To write OGG odbc.ini file :

[ODBC Data Sources] 
PGDSN1=DataDirect 14 PostgreSQL Wire Protocol 
PGDSN2=DataDirect 11 PostgreSQL Wire Protocol 

[ODBC] 
IANAAppCodePage=106 
InstallDir=/ogg21/ogg_ma 

[PGDSN1] 
Driver=/ogg21/ogg_ma/lib/GGpsql25.so 
Description=DataDirect 14 PostgreSQL Wire Protocol 
Database=PGSOUR
HostName=192.168.0.10
PortNumber=5432 
LogonID=postgres 
Password=Kunluntest

[PGDSN2] 
Driver=/ogg21/ogg_ma/lib/GGpsql25.so 
Description=DataDirect 11 PostgreSQL Wire Protocol 
Database=KunlunDB 
HostName=192.168.0.12 
PortNumber=5401 
LogonID=postgres 
Password=Kunluntest


2.3.3  Configure replication environment

Create identification and checkpoint tables ：

Sign in :

2.3.4  To configure & Start extraction process

Extract process configuration file ：

extract ext1 
SETENV(PGCLIENTENCODING = "UTF8" ) 
SETENV(ODBCINI="/ogg21/ogg_ma/odbc.ini" ) 
SOURCEDB PGDSN1 USERIDALIAS PG1, DOMAIN OGGMA 
exttrail ./dirdat/e1 
IGNOREREPLICATES 
TRANLOGOPTIONS FILTERTABLE ogg.ckpt 
table public.*;


2.3.5  To configure & Start the replication process

Copy process profile ：

replicat rep1 
SETENV(PGCLIENTENCODING = "UTF8" ) 
SETENV(ODBCINI="/ogg21/ogg_ma/odbc.ini" ) 
SETENV(NLS_LANG="AMERICAN_AMERICA.AL32UTF8") 
TARGETDB PGDSN2 USERIDALIAS PG2, DOMAIN OGGMA 
map public.*, target public.*;


2.3.6  Perform synchronization and check synchronization

You can insert data into the source database , Check the results in the target database :

You can also view the replication progress through the management interface  :

2.3.7  Pressure test

The stress test consists of two parts （ The machine configuration is low due to the virtual machine environment tested , The data volume of pressure measurement is relatively small ）

Source end pass sysbench Yes postgresql Real time data insertion ,  Check the target end kunlundb Update of data .

The target database runs under a certain load .

The source side performs data loading ：

sysbench oltp_read_write --table-size=100000 --tables=10 --pgsql-host=192.168.0.10 --pgsql-port=5432 --pgsql-user=postgres --pgsql-password=Kunluntest --pgsql-db=PGSOURE --db-ps-mode=disable --threads=1 --db-driver=pgsql prepare


Execute the pressure test script for the target database ：

sysbench oltp_read_write --table-size=100 --tables=10 --pgsql-host=192.168.0.12 --pgsql-port=5401 --pgsql-user=abc --pgsql-password=abc --pgsql-db=postgres --db-ps-mode=disable --threads=10 --db-driver=pgsql --time=600 run


Check the data synchronization at the target end ：

select count(*) from sbtest10;
 count
-------
 100000
(1 row)


Test conclusion ： adopt

END

Kunlun database is a HTAP NewSQL Distributed database management system , It can meet the all-round needs of users for the storage, management and utilization of massive relational data . Application developers and DBA The experience of using Kunlun database is similar to that of single machine MySQL And stand alone PostgreSQL almost the same , Because first of all, Kunlun database supports PostgreSQL and MySQL Double agreement , Support standards SQL:2011 Of  DML  Grammar and function and PostgreSQL and MySQL To the standard  SQL An extension of . meanwhile , Kunlun database cluster supports horizontal elastic expansion , Automatic data splitting , Distributed transaction processing and distributed query processing , Robust fault tolerance , Improve the intuitive monitoring, analysis and alarm capability , Cluster data backup and recovery   frequently-used DBA  Data management and operation . All these functions do not require any coding work on the application system side , No need DBA Artificial intervention , Non-stop service does not affect the normal operation of business . Kunlun database has a comprehensive OLAP  Data analysis capabilities , Passed TPC-H and TPC-DS Standard test set , It can analyze the latest business data in real time , Help users discover the value of data . Kunlun database supports the deployment of public cloud and private cloud environments , It can be done with docker,k8s Wait for cloud infrastructure to work seamlessly , You can easily build cloud database services . Please visit  

http://www.zettadb.com/

  Get more information and download Kunlun database software 、 Documents and materials .KunlunBase The project is open source

【GitHub：】

https://github.com/zettadb

【Gitee：】

https://gitee.com/zettadb