Problems and precautions about using data pumps (expdp, impdp) to export and import large capacity tables in Oracle migration

survey

I'm going to be here once recently Oracle data （Linux） transfer , Because the source database has a large amount of data , Therefore, the determined implementation scheme is as follows ：
1. Source library Oracle Users use data pump tools expdp Whole database （ user ） export .
2. Target library usage impdp Import .
3. Use kettle The tool synchronizes all tables with data changes .
4. Determine the service data source time , Stop the service connection in the source library , Perform the last data synchronization at the same time , Switch the relevant service data source to the target database . Finally, start the service to complete the migration .
In the use of expdp Specify concurrency as 10, Export in two parts , Because the data volume of individual tables in the source database is hundreds of millions to more than a billion , And more than 2T Picture class of LOB Data additionally specifies the table name to export ：
export 1
expdp username/password directory=expdp_dir dumpfile=2022xxxx_username_%U.dmp logfile=2022xxxx_username.log exclude=TABLE:“IN(‘TB_XX01’,‘TB_XX02’,‘TB_XX03’,‘TB_XX04’)” parallel=10
export 2
expdp username/password directory=expdp_dir dumpfile=TP2022xxxx_username_%U.dmp logfile=2022xxxx_username.log TABLES=‘TB_XX01’,‘TB_XX02’,‘TB_XX03’,‘TB_XX04’ parallel=10

What happened

When importing large tables of image type and billions of text type data, it is necessary to find the temporary table space of the database during the import process temp And fallback tablespaces undotbs1 The use of the Internet continues to increase , And failed to import a table for several days . Stopped importing after finding the problem , But it caused undo The large occupation of table space has not been released , It is found by inquiry that undo There are a lot of unfinished transactions in , And transactions are all transactions related to maintaining the primary key in the data warehousing of the primary key existing in the table at the time of import .
In fact, all indexes, constraints and triggers on the table except the primary key have been deleted and disabled before import , But I didn't expect that just maintaining the primary key led to so many undo data .

Processing mode

After finding the cause of the problem , All primary keys on the table have also been deleted , And the parameters are executed during import （undo If the tablespace is not released for a long time, please refer to the... At the end of the blog appendix To operate ）：
impdp username/password directory=expdp_dir dumpfile=ZP2022xxxx_username_%U.dmp logfile=impdp_username_2022xxxx.log TABLES=‘TB_XX01’,‘TB_XX02’,‘TB_XX03’,‘TB_XX04’ parallel=10 access_method=direct_path
This import was completed in a short time , And did not produce a lot of undo data .

summary

1. When processing large capacity data tables and using data pump export , It is recommended that the server's cpu Check the number , Set the parallelism properly , Speed up export . And exported dmp Also need to configure %U, Write concurrently with merge rows dmp file .
2. In import city , For large tables, it is recommended to clear all indexes and constraints on the table if unnecessary , Include primary key , Otherwise, a lot of undo data , Occupy undo Table space . Cause a large number of transactions to accumulate , Table data import is extremely slow .
3.impdp Export parameters for access_method=direct_path The check of database metadata will be ignored , Although it will speed up the import , But there is also a certain probability of error reporting , We should use it with caution .

appendix

When stopping import ,undo Tablespaces have produced a large number of undo data , And this time, because of a large amount of data import , So the archive was not opened , It also leads to undo The data is in redo Data coverage after log cycle , Unable to release for a long time undo data , And even set a new default table space UNDOTBS2 after , Still can't delete the old UNDOTBS1 Table space , And you will find that the database operation will be very catchy , At this time, because the database is still dealing with the old undo Transaction data in tablespace . Because there are also states in the table space ONLINE Fallback segment of , You need to wait for all the fallback segments OFFLINE Only then can UNDOTBS1 Table space delete , But because the archive was not opened , Some of the table space fallback segments cannot be automatically offline 了 . Check the status of the fallback section as follows ：
select * from dba_rollback_segs where tablespace_name=‘UNDOTBS1’

In order not to delay data migration , So it is handled in a special way .
1. Create a new undo Table space UNDOTBS2.
2. Default the database to undo The tablespace is specified as UNDOTBS2.
3. Force database shutdown shutdown abort . Because it's a database migration , It is not officially used, so it can be executed , But the production library should not do this , Forced stop will cause undotbs1 The data file of is damaged .
4. Open database , This startup is normal , But in the alter database open The phase lasts longer , see alter The log will find that the database scans the current redo, And perform a media recovery .
5. data open after ,sys Users log in to the database to view ：
select * from dba_rollback_segs where tablespace_name=‘UNDOTBS1’
You will find many old undo Table space UNDOTBS1 Many of the fallback states become “PARTLY AVAILABLE”, The status at this time indicates that the data file is damaged .
6. Record all status “PARTLY AVAILABLE” The name of the fallback segment of , That is to say SEGMENT_NAME.
7. establish pfile：create pfile=‘/home/oracle/initORCL.ora’ from spfile;
8. Close the database , You still need to force the shutdown shutdown abort. modify pfile, Set parameters ：
*._corrupted_rollback_segments=(_SYSSMU1${,}_{S}YSSMU2$,_SYSSMU3${,}_{S}YSSMU4$,_SYSSMU5${,}_{S}YSSMU6$,_SYSSMU7${,}_{S}YSSMU8$,_SYSSMU9${,}_{S}YSSMU10$)
（ notes ： Allow in rollback segments Start the database in case of corruption ）
The parameter value is the status “PARTLY AVAILABLE” The name of the fallback segment of .
Add parameters to pfile in .
9. Database to pfile start-up ：startup pfile=‘/home/oracle/initORCL.ora’;
10. There will still be a period of media recovery after the database is started , After fully opening , Can be deleted UNDOTBS1 Fallback tablespace .
11. After deleting the old tablespace , Close the database , It can be closed normally this time ：shutdown immediate. After closing, the database can be opened normally ：startup. Here we are undo Tablespace can be used normally .