Simple records of business system migration from Oracle to opengauss database

（ One ） Why move the database

As the world changes , No matter the country 、 Customer 、 Or the boss of the company .
All hope that the database can be controlled autonomously , Not American , Self developed , It's better to open source .
So use the 20 Year of Oracle The database is also ready to be replaced （ Omit roast , Regardless of workload ？）……

（1.1） About openGauss

Official website

From the PostgreSQL-XC, Led by Huawei 、 Jointly built by various ecological partners . The open source distribution agreement complies with Mulan loose License v2.
The current version 3.0.0, The introduction is divided into 【 Enterprise Edition 】 and 【 Lightweight version 】, But this time we use 【 Minimalist Edition 】……
The surrounding tools and ecology are as follows （ Omit roast ,Spark To do ？）：

（1.2） Database comparison and migration

Here is the website about openGauss Compared with the most popular relational database （ You can add SQL Server and PostgreSQL）
MySQL vs. OpenGauss vs. Oracle

Because work is used Oracle（ A few MySQL,SQL Server）.
No contact at all openGauss（ Not even used PostgreSQL） There is no time to study it slowly ,
So the general steps are as follows ：

• 1. Learn and install openGauss database .
• 2. Migration of business database structure and metadata .
• 3. Involving database software （Java） Development of .
• 4. Involving database software （Delphi/C++） Development of .
• 5. Involving database software （Go/Python） Development of .

（ Two ） Learn and install openGauss

（2.1） Official website documentation

Why did you even PostgreSQL All useless （ Omit roast ）……
contrast Oracle There are many more data types .
Many documents on the official website have not been updated with the version .
But the detailed steps and various materials are in Chinese ！

• https://blog.csdn.net/GaussDB
• https://www.modb.pro/openGauss
• bilibili

（2.2） Download the installation package and install

（2.2.1） Ready to install

• Get installation package ：openGauss-3.0.0-CentOS-64bit.tar.bz2 （ Minimalist Edition ）
• Unzip to directory ：tar -zxf /xxxx/openGauss-Lite-3.0.0-CentOS-x86_64.tar.gz -C ~/openGauss
• Check directory structure ：ll ~/openGauss/

 Total usage  16
drwxr-xr-x. 3 Shion Shion 4096 4 month    1 18:24 bin
drwx------. 3 Shion Shion   25 5 month    6 10:18 data
drwxr-xr-x. 3 Shion Shion   22 4 month    1 18:24 etc
drwxr-xr-x. 3 Shion Shion   24 4 month    1 18:24 include
drwxr-xr-x. 4 Shion Shion   95 4 month    1 18:24 jre
drwxr-xr-x. 5 Shion Shion 4096 4 month    1 18:24 lib
drwx------. 2 Shion Shion  126 5 month    6 10:18 logs
drwxr-xr-x. 5 Shion Shion   53 4 month    1 18:24 share
drwxr-xr-x. 2 Shion Shion   94 5 month    6 10:19 simpleInstall
-rw-r--r--. 1 Shion Shion   32 4 month    1 18:24 version.cfg

（2.2.2） Prepare the environment

It's on the official website , No more details ,CentOS7：

• Rely on checking ：
• libaio-devel Recommended version ：0.3.109-13
• flex Version required ：2.5.31 above
• bison Recommended version ：2.7-4
• ncurses-devel Recommended version ：5.9-13.20130511
• glibc-devel Recommended version ：2.17-111
• patch Recommended version ：2.7.1-10
• redhat-lsb-core Recommended version ：4.1
• readline-devel Recommended version ：7.0-13
  ——
• Turn off firewall
• Set character set parameters
• Set the time zone and time
• close swap Swap memory （ Optional ）
• close RemoveIPC

（2.2.3） install

The official example is 【 Minimalist Edition 】 No 【 Lightweight version 】, Then follow the tutorial .
And I'm not going to do it either , Build it quickly .

Into the simpleInstall Catalog .
$ sh install.sh -w " Your initial user password "

......
Would you like to create a demo database (yes/no)? yes   # Here no yes It's OK , No sample database generation .
Load demoDB [school,finance] success. 
...... 
[complete successfully]: You can start or stop the database server using:   
gs_ctl start|stop|restart -D $GAUSSHOME/data/single_node -Z single_node
......

Check ：
$ ps -ef |grep -v grep |grep gaussdb
$ gs_ctl query -D ~/openGauss/data/single_node

Errors should be reported during installation , Problems and solutions are mentioned on the official website .

After completion, some environment variables will be modified ,
But it changed after I installed it .bashrc Inside limit -n 1000000 Will report a mistake .

（2.3） Client and connection database

After the database is installed , The default build name is postgres The database of .

（2.3.1） Use gsql Command line connection

$ gsql -d postgres (-p 5432)

gsql ((openGauss 3.0.0 build 02c14696) compiled at 2022-04-01 18:12:34 commit 0 last mr  )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.

openGauss=#

（2.3.2）Navicat

You can't use the command line all at ordinary times , There must be a normal graphical interface management tool .
Oracle It's all for Toad…… I'm going to try this time Navicat.

But newly installed openGauss By default, you cannot connect to the database from another host , need ：

>> To configure pg_hba.conf Allow host connections ：

host    all             all             0.0.0.0/0                md5   # Make any address available md5 Encryption authentication method 

>> To configure postgresql.conf monitor / Binding address , And password encryption ：

listen_addresses = '*'                  # what IP address(es) to listen on;
local_bind_address = '0.0.0.0'             
password_encryption_type = 0            #Password storage type, 0 is md5 for PG, 1 is sha256 + md5, 2 is sha256 only

>> New database user ：
After configuring the above two items, restart the database ：

$ gs_ctl restart -D $GAUSSHOME/data/single_node -Z single_node

For security reasons, the initial user cannot access from the network , So create a new user and grant permissions .

$ gsql -d postgres

openGauss=# CREATE USER ac_man_parse PASSWORD ' New user's password ';
NOTICE:  The encrypted password contains MD5 ciphertext, which is not secure.
CREATE ROLE

openGauss=# ALTER ROLE ac_man_parse SYSADMIN;
ALTER ROLE

>> stay Navicat Middle configuration PostgreSQL Mode of access ：

episode ：

Windows10 The built-in firewall under , There is no problem that the program on the disk partition with the assigned drive letter passes .
But mount to the disk partition of the empty directory , The above program setting rules are useless , The built-in firewall still blocks , The principle is unknown .
So I received a lot socket error 10013, I thought it was a server problem .

（2.3.3）Data Studio

The download page
I don't know why I skipped the official tool , Just tried ,JDK persuade sb. to resign from an official position .
although Data Studio Of readme.txt It says ：

Software:  
----------------------------------------------------------------------
Java 1.8.0_181 or later    

But the actual operation is not good , So it doesn't continue to be used .

（2.3.3）Toad Edge

Trying , It's a little heavy .
The reaction is slow , But the operation logic and TOAD for Oracle almost .

Uh Freeware Version requires every 30 Log in to the account once a day for authorization , There is no solution for non networked machines （ No, Offline Pattern ）.
So I still give up , Wait and see if you want to buy a commercial version .

It seems to be able to support PostgreSQL Client tools , All can be connected openGauss Well .

（ 3、 ... and ） Business database structure and metadata migration

Official tools such as Ora2Pg It's not used yet .
but Navicat It is convenient to migrate data , Include Blob Type field data . You can try to import data without buying .

（3.1） Use tools to migrate data

1） function Navicat Menu Data Transfer, Select source and target ：

2） set an option ：
I'm testing Oracle->PostgreSQL, Options in the red box below （ In hexadecimal format BLOB） The tick of should be cancelled .
And fall again PostgreSQL->PostgreSQL, Then tick …… The principle is unknown ……

3） Select object ：
Select the data type and specific object to be migrated , If it is a similar database , Index 、 Sequence, etc .

4） Confirm and start ：
After dot , You can see the progress （ I have directed , There is no mapping here ）.
Don't close it after completion . You can go back and select another database to synchronize .

5） Check the data structure ：
After the migration is successful, you need to check ： Are the lengths of various data types correct , Big object （BLOB） Whether the content is correct .

（3.2） Pay attention to table building and derivative syntax

Of course , We also need to create tables and import metadata normally .
It is convenient to execute scripts with graphical tools , The main problem is Oracle Table of SQL The statement needs to be modified , The import data statement also needs to be modified . similar ：

• Get rid of ： SET DEFINE OFF.
• Get rid of ： Create table Modification of NOCOMPRESS,NOCACHE
• modify ：VARCHAR2(100 BYTE) -> VARCHAR2(100)
• modify ：ADD (CONSTRAINT XXXXX PRIMARY KEY -> ADD PRIMARY KEY
• wait ……

Not quite understand ：PK and Unique The index of , Do you want to build it yourself ？（Oracle Unwanted ）
Checked the Internet and said PostgreSQL It will also be built automatically , But I can't see it ？

（3.3） Annotation of tables and fields

Comments on tables and fields , Unable to join Oracle&MySQL Write directly to the table creation statement ,
Of course Oracle It also supports this method of creating tables before commenting , So it is more convenient to write the script in a general way . as follows ：

1. Build a watch first ：CREATE TABLE SampleTable ...
2. Note sheet ：COMMENT ON TABLE SampleTable IS ' Notes to the table '
3. Loop comment field COMMENT ON COLUMN SampleTable.Field1 IS ' Comment description for field '

（3.4） Coding leads to length differences

Created from the above steps openGauss The database character code is UTF8,
But the original Oracle The database is ZHS16GBK.
When importing data , If the field content is Chinese （ Multibyte ） Of , Maybe it used to be enough, but now it's too long .

️ Be careful ： Inaccurate --> Field approximate estimation GBK The length of Chinese characters x2, and UTF8 The length of Chinese characters x3.
See this article specifically 《【 Kunjin copy 】 The story of ： Talk about Chinese character coding and common character sets 》 Well ,
I won't repeat it here .

（ Four ） Yard farmer's work （monkey is coding）

（4.1）Java

Definitely with JDBC 了 ,
Just change a little , Other uses and Oracle equally （ Of course, pay attention SQL grammar , Don't use Oracle Special functions , Omit roast ）……

（4.1.1）PostgreSQL JDBC Driver

Use PostgreSQL Packages can also be accessed openGauss：

        <!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>42.3.5</version>
        </dependency>

	JDBCUrl = "jdbc:postgresql://host:port/db";
	driverClass = "org.postgresql.Driver";

（4.1.2）OpenGauss JDBC Driver

Use openGauss Package mode ：
This official document seems to have forgotten to update , Write in one place com.huawei.opengauss, One place is still written postgre.

        <!-- https://mvnrepository.com/artifact/org.opengauss/opengauss-jdbc -->
        <dependency>
            <groupId>org.opengauss</groupId>
            <artifactId>opengauss-jdbc</artifactId>
            <version>3.0.0</version>
        </dependency>

	JDBCUrl = "jdbc:opengauss://host:port/db";
	driverClass = "org.opengauss.Driver";

This way will be in console Print some logs , Our system needs to be removed .
I lost one log4j.properties go in , Find that kissing is not used log4j The output log ……
So set loggerLevel=off：

		...
		Properties info = new Properties();
		info.setProperty("loggerLevel","off");
		info.setProperty("user", Username);
		info.setProperty("password", gPass);		
		...
		try (Connection conn = DriverManager.getConnection(jdbcUrl, info)) {
    
			...
		}

（4.1.3）SQL Loader & COPY FROM STDIN

>>Oracle
When a large amount of data is put into storage ,Oracle It can be used SQL Loader：
Similar to the following command line （ No Java Can also call ）.
Parameters direct It is very important that high-speed import depends on it .
Try to be complete , If you encounter errors, you can see log and bad More convenient .

sqlldr user/[email protected] control=sample.ctl log=sample.log bad=sample.bad discard=sample.dis direct=true parallel=true streamsize=16777216

Of course , Control file in the command line （sample.ctl） It needs to be generated in advance , The contents are as follows ：
It's a little complicated , For example, the long field needs to specify the length .

OPTIONS (skip=0)
LOAD DATA CHARACTERSET ZHS16GBK INFILE 'Sample.txt'
APPEND INTO TABLE SampleTable
FIELDS TERMINATED BY X'09'
OPTIONALLY  ENCLOSED BY "'" 
TRAILING NULLCOLS (field1,field2 char(3000),field3, ......)

>>PostgreSQL（openGauss）
its JDBC Provides a very similar method , It's a little simpler .
Copy data from file to database table , as follows .
Not many parameters , Return the number of entries in the warehouse .

...
import org.opengauss.copy.CopyManager;
import org.opengauss.core.BaseConnection;
...
    public static long copyFromFile(Connection connection, String filePath, String tableName) throws SQLException, IOException {
    
        try (FileInputStream fileInputStream = new FileInputStream(filePath)){
    
            CopyManager copyManager = new CopyManager((BaseConnection)connection);
            return copyManager.copyIn("COPY " + tableName + " FROM STDIN with (DELIMITER '\t', ENCODING 'gb18030')", fileInputStream);
        }
    }

The separator and character set are replaced according to the actual situation .
Here's the odd thing Official document There is a grammatical problem in the example of , Missing comma , It's still very long ……

Try the main watch 50 About 10000 users , altogether 10 Import multiple tables , Including indexing ：
I feel no better than insert Faster sentences ……
If this speed is officially used ……ToT

• Oracle 12c: 20 second
• openGauss 3.0.0: 240 second

You can report an error if you try , such as ：extra data after last expected column.
It's not impossible to see where the problem is , There are too many text fields .

（4.2）C & C++

It's not yet in turn. ……

（4.3）Object Pascal

single 20 Year of delphi7 The old program has stopped maintenance , Are ready to give up .
adopt Java I found that since PostgreSQL You can access openGauss, Then try it too , It turns out that ：

	MainSession.Close;
	MainSession.ProviderName:='PostgreSQL';
	MainSession.Server:=theHost;
	MainSession.Port:=StrToIntDef(thePort,-1);
	MainSession.Database:=theDB;
	MainSession.Username:=Username;
	MainSession.Password:=gPass;
	MainSession.Open;

But we need to pay attention , host 、 port 、 Database name .
You need to separate and assign values like the above code .
Be careful not to talk to Oracle Just as direct MainSession.Server:=//ip:port/databasename.
In addition, it seems that you can't add parameters that the database doesn't support, such as MainSession.SpecificOptions.Values['xxxxx'] := 'yyyyy';
After all, it is not only supported Oracle A database .

️ ️ ️ The mystery of the bug️ ️ ️
【delphi+unidac+openGauss】, When SQL The selected field is of date type （Timestamp） when （ Including choosing now()）.
Program .AsDatetime Any date taken is always 2000-01-01 08:00:00, use .value It's the same thing , It has changed when you take it out .
Other language and database combinations have not found this problem .
I don't know. unidac The pot , still openGauss The pot .
at present I don't know how to solve , Temporarily change the program , Bypass and don't take the date directly .

（4.4） Case problem

Oracle By default, all object names are capitalized （ Unless you force lowercase with double quotation marks ）.
PostgreSQL Are all lowercase .

In a normal SQL There will be no problem in , But when it comes to selecting table names for comparison , The two databases are different .
It is best to Upper( Field ) To unify , Otherwise, it can only be in the program if else Judge whether different libraries write different SQL 了 .

also , Login user name Oracle Use any case ,
however PostgreSQL A strict distinction is needed .

（4.5） Unique syntax and highlighting

Not only the processing part needs to consider different database syntax .
Where there is a display or configuration interface , Pay attention to the unique keywords of different databases , Highlight .
such as Oracle Of `

select sysdate from dual;

such as PostgreSQL Of

select now();

---

（ 5、 ... and ） To be continued

【2022-05-07】 After a whole day, I can finally visit it simply openGauss 了 , Future work and problems , Continue to record .
【2022-05-10】 Added the problem of table and field annotation script , as well as sqlldr and COPY FROM STDIN Import file content .
【2022-05-13】 contrast sqlldr and COPY FROM STDIN efficiency , Report errors . Syntax and highlighting . mysterious bug. Client tools .

Actually do it SQL Many changes ……
Please refer to the next article for details ：《 Business system compatible database Oracle/PostgreSQL(openGauss)/MySQL The trivia of 》

---