当前位置:网站首页>Simple records of business system migration from Oracle to opengauss database
Simple records of business system migration from Oracle to opengauss database
2022-07-06 16:38:00 【Ruo Miaoshen】
List of articles
( One ) Why move the database
As the world changes , No matter the country 、 Customer 、 Or the boss of the company .
All hope that the database can be controlled autonomously , Not American , Self developed , It's better to open source .
So use the 20 Year of Oracle The database is also ready to be replaced ( Omit roast , Regardless of workload ?)……
(1.1) About openGauss
From the PostgreSQL-XC, Led by Huawei 、 Jointly built by various ecological partners . The open source distribution agreement complies with Mulan loose License v2.
The current version 3.0.0, The introduction is divided into 【 Enterprise Edition 】 and 【 Lightweight version 】, But this time we use 【 Minimalist Edition 】……
The surrounding tools and ecology are as follows ( Omit roast ,Spark To do ?):
(1.2) Database comparison and migration
Here is the website about openGauss Compared with the most popular relational database ( You can add SQL Server and PostgreSQL)
MySQL vs. OpenGauss vs. Oracle
Because work is used Oracle( A few MySQL,SQL Server).
No contact at all openGauss( Not even used PostgreSQL) There is no time to study it slowly ,
So the general steps are as follows :
- 1. Learn and install openGauss database .
- 2. Migration of business database structure and metadata .
- 3. Involving database software (Java) Development of .
- 4. Involving database software (Delphi/C++) Development of .
- 5. Involving database software (Go/Python) Development of .
( Two ) Learn and install openGauss
(2.1) Official website documentation
Why did you even PostgreSQL All useless ( Omit roast )……
contrast Oracle There are many more data types .
Many documents on the official website have not been updated with the version .
But the detailed steps and various materials are in Chinese !
(2.2) Download the installation package and install
(2.2.1) Ready to install
- Get installation package :
openGauss-3.0.0-CentOS-64bit.tar.bz2
( Minimalist Edition ) - Unzip to directory :
tar -zxf /xxxx/openGauss-Lite-3.0.0-CentOS-x86_64.tar.gz -C ~/openGauss
- Check directory structure :
ll ~/openGauss/
Total usage 16
drwxr-xr-x. 3 Shion Shion 4096 4 month 1 18:24 bin
drwx------. 3 Shion Shion 25 5 month 6 10:18 data
drwxr-xr-x. 3 Shion Shion 22 4 month 1 18:24 etc
drwxr-xr-x. 3 Shion Shion 24 4 month 1 18:24 include
drwxr-xr-x. 4 Shion Shion 95 4 month 1 18:24 jre
drwxr-xr-x. 5 Shion Shion 4096 4 month 1 18:24 lib
drwx------. 2 Shion Shion 126 5 month 6 10:18 logs
drwxr-xr-x. 5 Shion Shion 53 4 month 1 18:24 share
drwxr-xr-x. 2 Shion Shion 94 5 month 6 10:19 simpleInstall
-rw-r--r--. 1 Shion Shion 32 4 month 1 18:24 version.cfg
(2.2.2) Prepare the environment
It's on the official website , No more details ,CentOS7:
- Rely on checking :
libaio-devel
Recommended version :0.3.109-13
flex
Version required :2.5.31
abovebison
Recommended version :2.7-4
ncurses-devel
Recommended version :5.9-13.20130511
glibc-devel
Recommended version :2.17-111
patch
Recommended version :2.7.1-10
redhat-lsb-core
Recommended version :4.1
readline-devel
Recommended version :7.0-13
——- Turn off firewall
- Set character set parameters
- Set the time zone and time
- close swap Swap memory ( Optional )
- close RemoveIPC
(2.2.3) install
The official example is 【 Minimalist Edition 】 No 【 Lightweight version 】, Then follow the tutorial .
And I'm not going to do it either , Build it quickly .
Into the simpleInstall
Catalog .$ sh install.sh -w " Your initial user password "
......
Would you like to create a demo database (yes/no)? yes # Here no yes It's OK , No sample database generation .
Load demoDB [school,finance] success.
......
[complete successfully]: You can start or stop the database server using:
gs_ctl start|stop|restart -D $GAUSSHOME/data/single_node -Z single_node
......
Check :$ ps -ef |grep -v grep |grep gaussdb
$ gs_ctl query -D ~/openGauss/data/single_node
Errors should be reported during installation , Problems and solutions are mentioned on the official website .
After completion, some environment variables will be modified ,
But it changed after I installed it .bashrc
Inside limit -n 1000000
Will report a mistake .
(2.3) Client and connection database
After the database is installed , The default build name is postgres
The database of .
(2.3.1) Use gsql Command line connection
$ gsql -d postgres (-p 5432)
gsql ((openGauss 3.0.0 build 02c14696) compiled at 2022-04-01 18:12:34 commit 0 last mr )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.
openGauss=#
(2.3.2)Navicat
You can't use the command line all at ordinary times , There must be a normal graphical interface management tool .
Oracle It's all for Toad…… I'm going to try this time Navicat.
But newly installed openGauss By default, you cannot connect to the database from another host , need :
>> To configure pg_hba.conf Allow host connections :
host all all 0.0.0.0/0 md5 # Make any address available md5 Encryption authentication method
>> To configure postgresql.conf monitor / Binding address , And password encryption :
listen_addresses = '*' # what IP address(es) to listen on;
local_bind_address = '0.0.0.0'
password_encryption_type = 0 #Password storage type, 0 is md5 for PG, 1 is sha256 + md5, 2 is sha256 only
>> New database user :
After configuring the above two items, restart the database :
$ gs_ctl restart -D $GAUSSHOME/data/single_node -Z single_node
For security reasons, the initial user cannot access from the network , So create a new user and grant permissions .
$ gsql -d postgres
openGauss=# CREATE USER ac_man_parse PASSWORD ' New user's password ';
NOTICE: The encrypted password contains MD5 ciphertext, which is not secure.
CREATE ROLE
openGauss=# ALTER ROLE ac_man_parse SYSADMIN;
ALTER ROLE
>> stay Navicat Middle configuration PostgreSQL Mode of access :
episode :
Windows10 The built-in firewall under , There is no problem that the program on the disk partition with the assigned drive letter passes .
But mount to the disk partition of the empty directory , The above program setting rules are useless , The built-in firewall still blocks , The principle is unknown .
So I received a lot socket error 10013, I thought it was a server problem .
(2.3.3)Data Studio
The download page
I don't know why I skipped the official tool , Just tried ,JDK persuade sb. to resign from an official position .
although Data Studio Of readme.txt It says :
Software:
----------------------------------------------------------------------
Java 1.8.0_181 or later
But the actual operation is not good , So it doesn't continue to be used .
(2.3.3)Toad Edge
Trying , It's a little heavy .
The reaction is slow , But the operation logic and TOAD for Oracle almost .
Uh Freeware Version requires every 30 Log in to the account once a day for authorization , There is no solution for non networked machines ( No, Offline Pattern ).
So I still give up , Wait and see if you want to buy a commercial version .
It seems to be able to support PostgreSQL Client tools , All can be connected openGauss Well .
( 3、 ... and ) Business database structure and metadata migration
Official tools such as Ora2Pg
It's not used yet .
but Navicat It is convenient to migrate data , Include Blob Type field data . You can try to import data without buying .
(3.1) Use tools to migrate data
1) function Navicat Menu Data Transfer, Select source and target :
2) set an option :
I'm testing Oracle
->PostgreSQL
, Options in the red box below ( In hexadecimal format BLOB) The tick of should be cancelled .
And fall again PostgreSQL
->PostgreSQL
, Then tick …… The principle is unknown ……
3) Select object :
Select the data type and specific object to be migrated , If it is a similar database , Index 、 Sequence, etc .
4) Confirm and start :
After dot , You can see the progress ( I have directed , There is no mapping here ).
Don't close it after completion . You can go back and select another database to synchronize .
5) Check the data structure :
After the migration is successful, you need to check : Are the lengths of various data types correct , Big object (BLOB) Whether the content is correct .
(3.2) Pay attention to table building and derivative syntax
Of course , We also need to create tables and import metadata normally .
It is convenient to execute scripts with graphical tools , The main problem is Oracle Table of SQL The statement needs to be modified , The import data statement also needs to be modified . similar :
- Get rid of :
SET DEFINE OFF
. - Get rid of : Create table Modification of
NOCOMPRESS
,NOCACHE
- modify :
VARCHAR2(100 BYTE)
->VARCHAR2(100)
- modify :
ADD (CONSTRAINT XXXXX PRIMARY KEY
->ADD PRIMARY KEY
- wait ……
Not quite understand :PK and Unique The index of , Do you want to build it yourself ?(Oracle Unwanted )
Checked the Internet and said PostgreSQL It will also be built automatically , But I can't see it ?
(3.3) Annotation of tables and fields
Comments on tables and fields , Unable to join Oracle&MySQL Write directly to the table creation statement ,
Of course Oracle It also supports this method of creating tables before commenting , So it is more convenient to write the script in a general way . as follows :
- Build a watch first :
CREATE TABLE SampleTable ...
- Note sheet :
COMMENT ON TABLE SampleTable IS ' Notes to the table '
- Loop comment field
COMMENT ON COLUMN SampleTable.Field1 IS ' Comment description for field '
(3.4) Coding leads to length differences
Created from the above steps openGauss The database character code is UTF8
,
But the original Oracle The database is ZHS16GBK
.
When importing data , If the field content is Chinese ( Multibyte ) Of , Maybe it used to be enough, but now it's too long .
️ Be careful : Inaccurate --> Field approximate estimation GBK
The length of Chinese characters x2, and UTF8
The length of Chinese characters x3.
See this article specifically 《【 Kunjin copy 】 The story of : Talk about Chinese character coding and common character sets 》 Well ,
I won't repeat it here .
( Four ) Yard farmer's work (monkey is coding)
(4.1)Java
Definitely with JDBC 了 ,
Just change a little , Other uses and Oracle equally ( Of course, pay attention SQL grammar , Don't use Oracle Special functions , Omit roast )……
(4.1.1)PostgreSQL JDBC Driver
Use PostgreSQL Packages can also be accessed openGauss:
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>42.3.5</version>
</dependency>
JDBCUrl = "jdbc:postgresql://host:port/db";
driverClass = "org.postgresql.Driver";
(4.1.2)OpenGauss JDBC Driver
Use openGauss Package mode :
This official document seems to have forgotten to update , Write in one place com.huawei.opengauss, One place is still written postgre.
<!-- https://mvnrepository.com/artifact/org.opengauss/opengauss-jdbc -->
<dependency>
<groupId>org.opengauss</groupId>
<artifactId>opengauss-jdbc</artifactId>
<version>3.0.0</version>
</dependency>
JDBCUrl = "jdbc:opengauss://host:port/db";
driverClass = "org.opengauss.Driver";
This way will be in console Print some logs , Our system needs to be removed .
I lost one log4j.properties go in , Find that kissing is not used log4j The output log ……
So set loggerLevel
=off
:
...
Properties info = new Properties();
info.setProperty("loggerLevel","off");
info.setProperty("user", Username);
info.setProperty("password", gPass);
...
try (Connection conn = DriverManager.getConnection(jdbcUrl, info)) {
...
}
(4.1.3)SQL Loader & COPY FROM STDIN
>>Oracle
When a large amount of data is put into storage ,Oracle It can be used SQL Loader:
Similar to the following command line ( No Java Can also call ).
Parameters direct
It is very important that high-speed import depends on it .
Try to be complete , If you encounter errors, you can see log
and bad
More convenient .
sqlldr user/[email protected] control=sample.ctl log=sample.log bad=sample.bad discard=sample.dis direct=true parallel=true streamsize=16777216
Of course , Control file in the command line (sample.ctl) It needs to be generated in advance , The contents are as follows :
It's a little complicated , For example, the long field needs to specify the length .
OPTIONS (skip=0)
LOAD DATA CHARACTERSET ZHS16GBK INFILE 'Sample.txt'
APPEND INTO TABLE SampleTable
FIELDS TERMINATED BY X'09'
OPTIONALLY ENCLOSED BY "'"
TRAILING NULLCOLS (field1,field2 char(3000),field3, ......)
>>PostgreSQL(openGauss)
its JDBC Provides a very similar method , It's a little simpler .
Copy data from file to database table , as follows .
Not many parameters , Return the number of entries in the warehouse .
...
import org.opengauss.copy.CopyManager;
import org.opengauss.core.BaseConnection;
...
public static long copyFromFile(Connection connection, String filePath, String tableName) throws SQLException, IOException {
try (FileInputStream fileInputStream = new FileInputStream(filePath)){
CopyManager copyManager = new CopyManager((BaseConnection)connection);
return copyManager.copyIn("COPY " + tableName + " FROM STDIN with (DELIMITER '\t', ENCODING 'gb18030')", fileInputStream);
}
}
The separator and character set are replaced according to the actual situation .
Here's the odd thing Official document There is a grammatical problem in the example of , Missing comma , It's still very long ……
Try the main watch 50 About 10000 users , altogether 10 Import multiple tables , Including indexing :
I feel no better than insert Faster sentences ……
If this speed is officially used ……ToT
- Oracle 12c:
20
second - openGauss 3.0.0:
240
second
You can report an error if you try , such as :extra data after last expected column
.
It's not impossible to see where the problem is , There are too many text fields .
(4.2)C & C++
It's not yet in turn. ……
(4.3)Object Pascal
single 20 Year of delphi7 The old program has stopped maintenance , Are ready to give up .
adopt Java I found that since PostgreSQL You can access openGauss, Then try it too , It turns out that :
MainSession.Close;
MainSession.ProviderName:='PostgreSQL';
MainSession.Server:=theHost;
MainSession.Port:=StrToIntDef(thePort,-1);
MainSession.Database:=theDB;
MainSession.Username:=Username;
MainSession.Password:=gPass;
MainSession.Open;
But we need to pay attention , host 、 port 、 Database name .
You need to separate and assign values like the above code .
Be careful not to talk to Oracle Just as direct MainSession.Server:=//ip:port/databasename
.
In addition, it seems that you can't add parameters that the database doesn't support, such as MainSession.SpecificOptions.Values['xxxxx'] := 'yyyyy';
After all, it is not only supported Oracle A database .
️ ️ ️ The mystery of the bug️ ️ ️
【delphi+unidac+openGauss】, When SQL The selected field is of date type (Timestamp) when ( Including choosing now()
).
Program .AsDatetime
Any date taken is always 2000-01-01 08:00:00
, use .value
It's the same thing , It has changed when you take it out .
Other language and database combinations have not found this problem .
I don't know. unidac The pot , still openGauss The pot .
at present I don't know how to solve , Temporarily change the program , Bypass and don't take the date directly .
(4.4) Case problem
Oracle By default, all object names are capitalized ( Unless you force lowercase with double quotation marks ).
PostgreSQL Are all lowercase .
In a normal SQL There will be no problem in , But when it comes to selecting table names for comparison , The two databases are different .
It is best to Upper( Field )
To unify , Otherwise, it can only be in the program if else
Judge whether different libraries write different SQL 了 .
also , Login user name Oracle Use any case ,
however PostgreSQL A strict distinction is needed .
(4.5) Unique syntax and highlighting
Not only the processing part needs to consider different database syntax .
Where there is a display or configuration interface , Pay attention to the unique keywords of different databases , Highlight .
such as Oracle
Of `
select sysdate from dual;
such as PostgreSQL
Of
select now();
( 5、 ... and ) To be continued
【2022-05-07】 After a whole day, I can finally visit it simply openGauss 了 , Future work and problems , Continue to record .
【2022-05-10】 Added the problem of table and field annotation script , as well as sqlldr
and COPY FROM STDIN
Import file content .
【2022-05-13】 contrast sqlldr
and COPY FROM STDIN
efficiency , Report errors . Syntax and highlighting . mysterious bug. Client tools .
Actually do it SQL Many changes ……
Please refer to the next article for details :《 Business system compatible database Oracle/PostgreSQL(openGauss)/MySQL The trivia of 》
边栏推荐
- Click QT button to switch qlineedit focus (including code)
- Li Kou: the 81st biweekly match
- Market trend report, technical innovation and market forecast of double-sided foam tape in China
- Educational Codeforces Round 130 (Rated for Div. 2)A~C
- Read and save zarr files
- Submit several problem records of spark application (sparklauncher with cluster deploy mode)
- 图图的学习笔记-进程
- Research Report on hearing health care equipment industry - market status analysis and development prospect prediction
- Acwing: Game 58 of the week
- 第7章 __consumer_offsets topic
猜你喜欢
Problem - 922D、Robot Vacuum Cleaner - Codeforces
Codeforces Round #797 (Div. 3)无F
解决Intel12代酷睿CPU单线程调度问题(二)
Pytorch extract skeleton (differentiable)
Log statistics (double pointer)
Solve the problem of intel12 generation core CPU [small core full, large core onlookers] (win11)
Installation and use of VMware Tools and open VM tools: solve the problems of incomplete screen and unable to transfer files of virtual machines
<li>圆点样式 list-style-type
The concept of spark independent cluster worker and executor
Story of [Kun Jintong]: talk about Chinese character coding and common character sets
随机推荐
Chapter III principles of MapReduce framework
Advancedinstaller installation package custom action open file
本地可视化工具连接阿里云centOS服务器的redis
Research Report of desktop clinical chemical analyzer industry - market status analysis and development prospect prediction
Generate random password / verification code
Research Report on market supply and demand and strategy of Chinese table lamp industry
Codeforces round 797 (Div. 3) no f
Chapter 7__ consumer_ offsets topic
Is the sanic asynchronous framework really so strong? Find truth in practice
Chapter 1 overview of MapReduce
图像处理一百题(11-20)
Share an example of running dash application in raspberry pie.
(lightoj - 1323) billiard balls (thinking)
Chapter 6 datanode
新手必会的静态站点生成器——Gridsome
QT实现窗口置顶、置顶状态切换、多窗口置顶优先关系
Summary of game theory
Classic application of stack -- bracket matching problem
第5章 消费者组详解
Browser print margin, default / borderless, full 1 page A4