Jincang database KingbaseES V8 GIS data migration solution (3. Data migration based on ArcGIS platform to KES)

3. 基于ArcGISPlatform of data migration toKES

本章主要介绍基于ArcGIS/GeoScene Two kinds of data migration solution platform,首先是基于KDTS的迁移,Mainly used in the data layer and more massive data,另一种为利用ArcGIS/GeoScene Platform of heterogeneous database migration,Mainly some small layer or a small amount of data migration needs.

3.1. KDTS工具迁移GIS数据

3.1.1. KDTS迁移步骤

The following tells throughKDTS工具完成GISSpatial database migration work.

通过KDTS迁移GISData is mainly based on two things：

其一,在GIS平台中,数据存储在OracleSpatial数据库中,且数据量巨大.Mentioned in section in the previous chapterGISPlatform for migration elements such as class、要素集、A single library such as a small amount of data,When the data table size large,The migration process is slow;

其二,原始数据存储在OracleSpatialThe data in the database,No matter how much the amount of data,但没有GIS平台支撑,Only backup of the database file.

上述两种情况下,都可以采用KDTS数据迁移工具完成GISSpatial database migration work.

图 3.1.1 KDTS迁移方案

1. 以迁移OracleThe database data, for example（The default client installedOracle软件）

   1. ArcGISConnected to the source database and target database

   • 创建数据库连接

     在开始菜单中启动ArcMap软件,在菜单栏点击“Arctoolbox”图标,On the right shows theArctoolbox栏中,Select the data management tool->地理数据库管理->创建企业级地理数据库,界面截图如下：

     

   • Oracle数据库（待迁移GIS数据的数据库）连接配置

     

     图 3.1.2 数据库配置

   • KES数据库连接配置

     通过ArcMap菜单栏,打开ArcCatalog软件,In a directory tree click“数据库连接”->“添加 数据库连接”.在“数据库连接”对话框中设置KES连接参数,其中需要注意的是instance端口号不使用5432时,Need to indicate the port number;例如：192.168.4.30,5444.

     

     图 3.1.3 数据库连接

   • Start the geographical database

     在ArcCatalogSoftware in the directory tree click the connected database,右键选择“Start the geographical database”功能,The source database and target database to do this.Configuration corresponding parameters,选择ArcGISThe authorization documents need to pay attention to only use*.ecp文件.

2. KDTS迁移

KDTSThe migration tool use of plug-in dynamic loading to migrate the database access interface,方便用户定制和使用.支持同异构数据源之间的数据迁移、支持结构迁移、支持全量数据迁移、支持列名映射,支持数据迁移过滤,在配置数据任务时,可以对迁移的表配置 where 条件、通过匹配的 where 条件过滤需要迁移的数据.

The data transfer between heterogeneous data sources：支持 Oracle9i、10g、11g、12c、19c 到 KingbaseES V8 的数据库迁移.（KingbaseES V8.7 The migration tool also supportsmysql5.1、sqlserver2005、sqlserver2014、神通、sybase15.7 到 KingbaseES V8 The table structure and data migration.

下图为KDTSData migration fields corresponding to the table,其中包含GIS常用的模型ST_GEOMETRY和SDO_GEOMETRY.

图 3.1.4 KDTSMigration program Settings

下面详细介绍KDTSThe use of the migration tool plan：

1. 启动数据迁移工具

选择『开始』|『程序』|『KingbaseES V8』|『数据迁移工具』,启动数据迁移工具.

1. Create a data migration project

Open data migration tools,Tool has created a default project.Users can also right click on the left side resource manager new data migration project.Data migration project is the starting point of data migration,A migration project can contain more than one database connection、Multiple data migration tasks and migrated log information.如图下图所示.

Perform a migration task,The necessary steps are as follows：

• New source of library database connection,New purpose library database connection;

• 新建迁移任务,Select the source library and purpose;

• Open the migration task,Select the table object configuration;

• On source-sink relationship with the object library mapping;

• 执行迁移任务,See the migration log;

图 3.1.5 新建项目

KingbaseES V8 Database data migration tools usingJDBC The way to manage database connection,支持JDBC URL 参数配置.可以在资源管理器上对数据库连接进行删除或者编辑操作,The database connection information persistent local,迁移时不需要重复配置连接.

The database connection under the data migration project list,Right-click to create a new database connection,弹出下图所示窗口,填写相应信息,创建数据库连接.

图 3.1.6 创建数据库连接

1. 创建迁移任务

Under the background of the data migration project【Database migration task】Node right click create migration task,Migration task need to select the already created the source database connection and the target database connection,完成输入后点击【确定】按钮,Create success after will automatically open the migration task editor.The migration task editor interface as shown in the figure below：

图 3.1.7 新建迁移任务

图 3.1.8 The migration task editor interface

1. The table object configuration

After the new migration task,Double-click the source sink node to open“非表对象”设置,Contains the view、索引、外键、缺省、触发器、检查约束、唯一约束、存储过程、函数、程序包、序列、同义词、Comments were thirteen kinds of non table objects.Whether the user can choose in the migration task migration table objects of these.

如下图所示,The table object Settings,The default are not choose,Only migration table definitions and data table.此外,从 Oracle 到 KingbaseES 的迁移中,In addition to the table definition and table data,Users can also select view、索引、外键、缺省、检查约束、Uniqueness constraints and sequence of table objects such as automatic migration.

For the database stored procedure between、函数、Packages, and triggers migration,The transplantation of them need to KingbaseES And other grammatical differences between database manually change after,再在 KingbaseES Founded after the execution on the server. 需要注意的是,Don't have the selected table objects,Purpose to manually create all the database.

图 3.1.9 The table object Settings

1. Set the source database and target database relational mapping

This step is to choose to migration from the source database table, or other database object corresponding to the target database mode.双击【The migration task editor】The goal of library node,打开【The source database and target database relational mapping】,如下图所示,The target database node configuration requires the user to select tables or other database objects drag、双击、Right click to add to the target mode or database to complete the mapping relationship between.Under the target database show only from the source database drag the table or other database object,The target database before mapping operation,Display mode not only shows its own database objects.

图 3.1.10 The source database and target database relational mapping

1. Set table mapping relation

Double-click the object library table objects in the tree,可以进行【表映射关系】设置,In the dialog box can be【The column map selection】设置、【DDL】查看和【SELECT】自定义的操作,如下图所示.

图 3.1.11 Set table mapping relation

• 【The column map selection】Page you can set the source and target table column corresponding relationship,You can modify the target table column name、The target table column type、精度、标度、Is null and default value.

• 【DDL】Is a database tables are created on the target sql 语句,可以通过【The column map options】Page column information is changed automatically modify content.

• 【SELECT】Page for migrating to query the source table,可以对迁移的表配置 where 条件、通过匹配的 where 条件过滤需要迁移的数据.

1. 执行迁移任务

图 3.1.12 An overview of the data migration

Upon completion of the above configuration,Can be the source database to the target database migration,Open data migration task editor,点击工具栏上的【执行】按钮,The official start of the data migration.

The official start of the data migration with migrating object statistics list,The user can according to this list to determine whether a number of data correctly.如上图所示.在迁移结束后,In the console and log file statistics the migration results,Including tables and all database objects have chosen to transfer.

1. 至此,Complete use of the migration toolGIS数据迁移.

1. 案例说明

案例一

通过模拟10Provinces million level class diagram spot data,使用金仓KDTS工具完成GIS数据迁移,The migration time for1个小时左右.KDTSTool transfer efficiency than traditional method4到5倍.

3.1.2. The migration results show

1. A visual check

1. 显示验证：ArcMap软件中,The migrated data will drag the window.Data to demonstrate the diagram below：

图 3.1.13 数据显示

1. 操作验证：验证数据正确性,Using the migration data to increase、删、改、Check the function such as validation,Such as function analysis result judgment correctness.

2. 应用程序验证：Run with the migration data related to the application,验证数据正确性.

1. Database comparison check

KingbaseES V8After the migration will generate migration report,The report record migration object before and after the contrast results.After the completion of the operation to perform migration task,Generates the migration report.There are two kinds of migration report：Text report and histogram.如图下图所示.

图 3.1.14 迁移报告-文本报告

图 3.1.15 迁移报告-Histogram report

The above two kinds of data validation done,To ensure data migration work is done,The migration results right.

3.1.3. 常见问题汇总（FAQ）

1. 数据迁移中, Complex tables and constraint transfer failure problem.

答：在 KingbaseES 之间可以迁移的数据库对象包括：表、视图、存储过程、函数、程序包、触发器、索引、主外键、非空约束、检查约束、唯一约束、缺省值.若目的库中存在与要迁移的存储过程、函数、触发器或程序包相同的名称, Data migration tools will be deleted after being moved in.

其他数据库向 KingbaseES 可以迁移的数据库对象包括：表、索引、主外键、非空约束、检查约束、唯一约束、缺省值.不支持函数索引和表达式索引的迁移;不支持聚集索引的迁移.

1. In data migration data migration tools, Will other imported into the database table KingbaseES V8 数据库时,KingbaseES V8 The database and no and you want to import table namesake, table, Why would the migration log report name repetition error, Leading to the migration failed？

答：在 KingbaseES V8 In the same pattern in the table below、视图、Sequence all use the same namespace,即表、Between the view and the sequence can not wish,Otherwise will create table after covering the already existing table.

1. 源库为 KingbaseES V7,Migrating Chinese table Chinese garbled question.

答：编辑 KingbaES V7 的数据库连接,添加参数 clientencoding=UNICODE, 保存.Open the corresponding migration task,Refresh the source-sink node,即可.

1. 数据迁移中, Complex tables and constraint transfer failure problem.

答：当迁移 Oracle Complex tables and constraints,If always encounters an error,则可能是因为 KingbaseES 自带的 Oracle JDBC Driver version with Oracle JDBC 版本不兼容.此时,用户可用 Oracle 数据库安装目录 JDBC/lib 文件夹下的 JDBC 驱动程序,替换 $KINGBASE_HOME/plugins/database/Oracle 目录下的 JDBC 驱动程序,然后重新迁移.

Oracle 数据库向 KingbaseES 可以迁移的数据库对象包括：表、索引、主外键、非空约束、检查约束、唯一约束、缺省值.需要注意的是：不支持函数索引和表达式索引的迁移;不支持聚集索引的迁移.

1. 数据迁移中, 使用【追加数据】Way of migration will cause the failure of migration problem.

答：为了提高迁移速度,在【配置】对话框中,将【A batch submission record number】设置为 50000.When additional data,If submitted in a 50 条记录中,If you have some record's primary key value is same as the primary key of existing records,会导致 50 Record all the migration failed.因此在使用【追加数据】When migration way,建议将【A batch submission record number】设置为 1,So that we can make the primary key duplicate records the migration failed,The primary key values are not duplicate records migration success.

1. Data migration times wrong“Caused by: com.kingbase8.util.KSQLException: ERROR: value too long for type character varying(50)”

答：原因：The source library server parameter char_default_type=char,The target library kingbaseES V8 的服务器参数 char_default_type=byte,Part of the data migration to v8 After the overflow.

修改【首选项】 -【迁移配置】选项卡的“设置 char 默认类型”,选择“char”或者“Consistent with the source library”.Save to the migration error table after.

1. The migration tool can transfer thoseGIS数据类型？

答：In the process of migration Settings,Set migration data type, KDTSTools to moveST_GEOMETRY和SDO_GEOMETRY数据类型.其中ST_GEOMETRY为ArcGISPlatform of the generated data type,SDO_GEOMETRY为Oracle的数据类型.

3.2. ArcGIS/GeoScene平台迁移GIS数据到KES

3.2.1. 利用ArcGIS/GeoSceneThe software of data migration

This section mainly introduces the use ofArcGISSoftware toKES数据库迁移GIS数据的方案,Data migration needs to be installed beforeKGIS插件,安装方法参见《KingbaseGIS参考手册》.

1. Moving object that

• File geographic database and personal geographical database,即MDB或者GDB文件.

• 商用数据库：PostgreSQL、Oracle spatial、DM、MYSQL、SQL Server等.

1. 以迁移OracleThe database data, for example（The default client installedOracle软件）

   1. ArcGISConnected to the source database and target database

   • 创建数据库连接

     在开始菜单中启动ArcMap软件,在菜单栏点击“Arctoolbox”图标,On the right shows theArctoolbox栏中,Select the data management tool->地理数据库管理->创建企业级地理数据库,界面截图如下：

     

   • Oracle数据库（待迁移GIS数据的数据库）连接配置

     

     图 3.2.1 数据库配置

   • KES数据库连接配置

     通过ArcMap菜单栏,打开ArcCatalog软件,In a directory tree click“数据库连接”->“添加 数据库连接”.在“数据库连接”对话框中设置KES连接参数,其中需要注意的是instance端口号不使用5432时,Need to indicate the port number;例如：192.168.4.30,5444.

     

     图 3.2.2 数据库连接

   • Start the geographical database

     在ArcCatalogSoftware in the directory tree click the connected database,右键选择“Start the geographical database”功能,The source database and target database to do this.Configuration corresponding parameters,选择ArcGISThe authorization documents need to pay attention to only use*.ecp文件.

   1. 数据迁移

   • The operation of yi hasOracle和KESThe database at the same timeArcGIS平台连接,下面介绍如何将GIS数据迁移到KES数据库.

     After the connection database below：

     

     图 3.2.3 The data directory tree

     在OracleDatabase choose to migrate data elements set,右键copy;

     图 3.2.4 数据拷贝

   • 选择“数据库连接”In the connectedKES数据库,The right to build paste,弹出下图界面,选择ok;

     

     图 3.2.5 数据迁移

     According to data migration progress bar,如下图:

     After the success of the migration as shown in the figure below:

     完成数据库迁移.

   至此,完成基于ArcGIS/GeoScenePlatform of data migration.If you need to migrate the database table with a small amount of elements,Or data volume less space data can be in this way,But once the spatial data volume table many large space,建议使用KDTSThe migration tool batch migration data.

2. 案例介绍

案例一

1. Project application environment：ArcGIS,X86,Linux.

2. 数据迁移：Using the characteristic of the spatial data quantity offline migration tool,实现Oracle SpatialTo the National People's Congress Jin Cang spatial database of fast automatic smooth migration,48小时完成3TB+Oracle Spatial数据迁移.

3. 稳定性：Database concurrent connections1000+,Under pressure from big concurrent high throughput business,数据库稳定运行.

4. 性能：Provided by the optimization of the spatial data index,And depth of space function, optimization,Makes the spatial data processing performance improvement80%.

3.2.2. The migration results show

1）A visual check

1. 显示验证：ArcMap软件中,The migrated data will drag data window.Data to demonstrate the diagram below：

图 3.2.6 数据显示

1. 操作验证：验证数据正确性,在ArcMapSoftware using the migration data to increase、删、改、Check the function such as validation,Such as the accuracy analysis of the results of the validation functions.

2. 应用程序验证：Run with the migration data related to the application,验证数据正确性.

3.2.3. 常见问题汇总（FAQ）

1. What can currently migration types in the databaseGIS数据？

   答：Can migrate through theArcGIS存储的ST_GEOMETRY类型的数据,数据库类型不限.如PostgreSQL、Oracle spatial、DM、MYSQL、SQL Server等.