Before Oracle 19C migration, how important is it to do a good job of rat playback test?

The authors introduce

Liang mingtu , Chief architect of new torch network , More than ten years of database operation and maintenance 、 Database design 、 Data governance and system planning and construction experience , Have Oracle OCM、Togaf Enterprise architect （ Appraisal level ）、IBM CATE Etc , Once obtained dbaplus year MVP And Hua Weiyun MVP Such as the honor , And participated in the compilation of national standards for data asset management . In the database operation and maintenance management and architecture design 、 Operation and maintenance system planning 、 There is in-depth research on data asset management .

background

The core system database of a customer has always been IBM AIX Minicomputers and Oracle 11g R2, This database has been patched up for nearly a decade . Plans to move to x86 In the environment Oracle 19G, In previous migration tests and application tests , It's all about simple tests , Have not tried to simulate the pressure of the existing network . This may cause the application to connect to the new library , Unable to deal with all kinds of unknown problems in time .

So consider using RAT Medium Database Replay, Simulate the operation of the existing network , To discover data migration 、 Compatibility 、 Performance and so on .

RAT Introduce

Oracle Real Application Testing（ abbreviation RAT） yes 11gR2 One of the important feature, The original intention of its launch is to meet the needs of the data center after changes , There are good methods and tools to measure the impact of these changes on the application of the production environment , Better evaluate things like hardware upgrades 、 Software upgrade 、 The impact of architectural changes and so on on customer applications .

Real Application Testing There are actually two solutions , Namely Database Replay and SPA（SQL Performance Analyzer）. It used to be SPA I won't introduce you , This is mainly about RAT And the problems I encountered in the process of using .

Database Replay, As the name suggests, database playback can be understood as a video recorder , By recording the load generated by real-time applications on the source system , And get the changed environment to play , To make a comparison . Database playback is divided into four stages ： Recording 、 Preprocessing 、 The playback 、 Results analysis and reporting .

Test environment

• Source library host ：IBM 880
• Source library operating system ：AIX
• Source library version ：11.2.0.4.16
• whether RAC： yes
  
• Target library host ： domestic x86 The server
• operating system ：Redhat 7
• Target library version ：19.7
• whether RAC： yes

RAT Testing process

1、 Recording

1） Catalog preparation

First create an operating system directory , Used to store recorded files , Because it is RAC Environmental Science , Therefore, the requirement is a goal shared by two computers , For example, we use NFS The directory is attached to two nodes ：

Log in and copy

      
      

       
       ​mkdir -p /datadump/rat_test
      
      
      
      

       
       
1.
      
      

Create a... In the database DIRECTORY：

Log in and copy

      
      

       
       $ sqlplus / as sysdba
       
       
SQL> create directory DBRAT_0218 as '/home/oracle/rat_test';
       
       
SQL > grant read,write on directory DBRAT_0218 to public;
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
      
      

2） Set the conditions for capturing

Sometimes we need to limit the workload to capture only part , This setting can be used to achieve , In this test, we mainly capture the load of users in one of the cities ：

Log in and copy

      
      

       
       ​exec dbms_workload_capture.ADD_FILTER(fname =>'FILTER_SYS',fattribute => 'USER',fvalue => 'NGCRM_ST');
      
      
      
      

       
       
1.
      
      

3） Turn on capture

Log in and copy

      
      

       
       BEGIN
       
       
DBMS_WORKLOAD_CAPTURE.start_capture (name => 'DBREPLAY_0218',
       
       
                                      dir => 'DBRAT_0218',
       
       
                                 duration => 7200);
       
       
END;
       
       
/
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
      
      

• name Is the name of this capture task ;
• dir It was created above directory;
• duration It's capturing time , It has 7200, That is to catch 2 Hours , This parameter can be set to NULL, It's recording without time limit , Until a stop command is issued .

Log in and copy

      
      

       
       ​DBMS_WORKLOAD_CAPTURE.finish_capture;
      
      
      
      

       
       
1.
      
      

Problems encountered ： In the first contact and use DBReplay When , A capture command has been initiated in a library , Using a shared directory , Sometimes, no matter how initiated , After that 1-2 It's over in minutes （ There is no set capture time ）. At that time, it was thought that the business volume was too small and the process ended automatically , I didn't pay much attention to . Later, it is sometimes impossible to collect during busy business hours , Looking for all kinds of information, there is no relevant information . Later, I changed a directory and there was no such problem again .

4） Monitor capture status

Use the following command to check whether the recording is complete ：

Log in and copy

      
      

       
       select id,name,status,start_time,end_time,connects,user_calls,dir_path from dba_workload_captures;
      
      
      
      

       
       
1.
      
      

2、 The target side loads data

In order to simulate the pressure and feedback of the system more accurately , We need to load the data related to the existing database into the target database . Because it's a test environment , We'll collect the easiest Datapump The way the source database will be related to several schema Pour out , And load it to the target database . Space relationship , The operation process of this step is omitted .

3、 Load preprocessing

1） Load preprocessing

Before playing back the load , The load needs to be preprocessed .

Create a directory object to store the recorded files of source data , Because the target environment is also RAC, So you also need to use shared directories ：

Log in and copy

      
      

       
       SQL> create directory DBRAT_0218 as '/ACFS01/rat_0218';
       
       
SQL > grant read,write on directory DBRAT_0218 to public;
       
       
SQL> exec dbms_workload_replay.process_capture('DBRAT_0218');
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
      
      

Problems encountered ： In the initial test , We've been stuck in this step , The sentence has never been completed , At first, I thought this pretreatment would take a lot of time , So I kept waiting , It can't be done for hours or even a day , Never succeeded , And the background log has the following error ：

Log in and copy

      
      

       
       DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE:WARNING:Preprocessing encountered a CORRUPT file wrc_5ph12b0010fu.rec in /ACFS01/dbreplay/inst1/aa.
       
       
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE:WARNING:Preprocessing encountered a CORRUPT file wrc_5ph12b0010hr.rec in /ACFS01/dbreplay/inst1/aa.
       
       
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE:WARNING:Preprocessing encountered a CORRUPT file wrc_5ph12b0032sd.rec in /ACFS01/dbreplay/inst1/aa.
       
       
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE:WARNING:Preprocessing encountered a CORRUPT file wrc_5ph12b00154sb.rec in /ACFS01/dbreplay/inst1/aa.
       
       
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE:WARNING:Preprocessing encountered a CORRUPT file wrc_5ph12b0010fu.rec in /ACFS01/dbreplay/inst1/aa.
       
       
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE:WARNING:Preprocessing encountered a CORRUPT file wrc_5ph12b0010fu.rec in /ACFS01/dbreplay/inst1/aa.
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
       
       
5.
       
       
6.
      
      

Later submitted MOS, The official suggestion is to fix the source and target databases with corresponding patches bug.

Target library ：one-off-patch, Patch ：21117072, After this patch , The preprocessing command is no longer stuck waiting , If there is an error in the file, exit directly .

Source library ：one-off-patch, Patch ：17373277, Fixed the file error in the above figure bug.

2） Initialize playback

Log in and copy

      
      

       
       ​exec dbms_workload_replay.initialize_replay('DBREPLAY_0218','DBRAT_0218');
      
      
      
      

       
       
1.
      
      

PS： The first parameter is the name of the capture task , The second parameter is where the file is stored DIRECTORY.

3） Connection configuration mapping

Because the database configuration of playback environment and capture environment is different , A mapping process is needed .

Unmapped ：

Log in and copy

      
      

       
       ​select conn_id,capture_conn from dba_workload_connection_map order by 1;
      
      
      
      

       
       
1.
      
      

Log in and copy

      
      

       
       ​exec DBMS_WORKLOAD_REPLAY.REMAP_CONNECTION (connection_id =>  1 , replay_connection => '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=xxx.xxx.xxx.xxx)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME= xxxx )))');
      
      
      
      

       
       
1.
      
      

PS：connection_id Is the above statement to find out conn_id, Change as much as you can ,replay_connection It just needs to be modified IP、 Port and SERVICE_NAME that will do .

After mapping ：

4） Ready to play back

Log in and copy

      
      

       
       ​exec dbms_workload_replay.prepare_replay(synchronization=>FALSE);
      
      
      
      

       
       
1.
      
      

We use the default parameters here , If you need to increase or reduce the pressure of playback , You can configure connect_time_scale and think_time_scale These two parameters , The default values of these two parameters are 100（ percentage ）, That is to say 1:1 replay , If you put connect_time_scale Set to 50%, That is, the interval between two connections is shortened to the original 50%（10 It will be shortened to 5 minute ）, So as to increase the pressure on the database .

for example ：

Log in and copy

      
      

       
       ​exec dbms_workload_replay.prepare_replay(synchronization=>FALSE, connect_time_scale=>50, think_time_scale=>50);
      
      
      
      

       
       
1.
      
      

4、 replay

1） Prepare the client

Depending on the intensity of the workload , Prepare the corresponding client , have access to wrc To measure how many clients are needed .

Log in and copy

      
      

       
       ​wrc mode=calibrate replaydir=/ACFS01/rat_0218
      
      
      
      

       
       
1.
      
      

You can see that there is a sentence on it that is Consider using at least 30 clients divided among 8 CPU(s), Then we need to prepare at least 30 A client , Otherwise, there will be resource overflow and abnormal end .

At this time, the client is in the ready state , Waiting for the play to start （ Start at least 1 A client , Otherwise, you can't start playback ）.

Because of the need for 30 individual , So you can write a script on one machine for Cycle run 30 Background processes .

The reference script is as follows ：

Log in and copy

      
      

       
       for ((i=1;i<=30;i++)) 
       
       
do
       
       
nohup wrc system/[email protected]_ST mode=replay replaydir=/ACFS01/rat_0218 > /ACFS01/wrc_log/wrc_$i.log &
       
       
done
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
       
       
4.
      
      

2） Start playback

Log in and copy

      
      

       
       ​exec dbms_workload_replay.start_replay;
      
      
      
      

       
       
1.
      
      

Of course , We can call the following procedure through , Control playback job ：

Log in and copy

      
      

       
       DBMS_WORKLOAD_REPLAY.PAUSE_REPLAY (); ## Stop work 
       
       
DBMS_WORKLOAD_REPLAY.RESUME_REPLAY (); ## Keep working 
       
       
DBMS_WORKLOAD_REPLAY.CANCEL_REPLAY (); ## Cancel the assignment 
      
      
      
      

       
       
1.
       
       
2.
       
       
3.
      
      

3） Monitor playback

In a replay , Any mistake , Or there is a data discrepancy between the production environment and the test environment , Are recorded as divergent calls . Get information about these divergent calls , Include ：SQL ID、SQL Text 、 Binding variable values , You can use the view DBA_WORKLOAD_REPLAY_DIVERGENCE, as well as GET_DIVERGENT_STATEMENT obtain . for example ：

Log in and copy

      
      

       
       ​SQL> SELECT REPLAY_ID,STREAM_ID,CALL_COUNTER FROM DBA_WORKLOAD_REPLAY_DIVERGENCE;
      
      
      
      

       
       
1.
      
      

4） View playback Report ：

Log in and copy

      
      

       
       ​select dbms_workload_replay.report(139,'HTML') from dual;
      
      
      
      

       
       
1.
      
      

You can also export AWR Report on the same time period as the source system AWR The report compares .

Other information related to playback , You can refer to the following system view ：

• DBA_WORKLOAD_CAPTURES List all loads captured ;
• DBA_WORKLOAD_FILTERS List filters defined by all captured loads ;
• DBA_WORKLOAD_REPLAYS List all the loads that have been repeated ;
• DBA_WORKLOAD_REPLAY_DIVERGENCE List all divergent calls ;
• DBA_WORKLOAD_REPLAY_FILTER_SET List all the filters defined by the repeated load ;
• DBA_WORKLOAD_CONNECTION_MAP List the mapping information of all connection strings used for replay ;
• V$WORKLOAD_REPLAY_THREAD List the session information of all current replay clients .

Summary

Our customer's Oracle 11g to 19c Relocation project implementation , It actually took months , It has experienced many tests in different environments 、 Migration and cutover process . This test is an early part of this major project , After this test , Exposed our database parameters 、 Structure migration 、SQL Code and other aspects of the problem . Fortunately, , Early detection and early intervention and repair can be achieved through testing , The problem is not reserved for system cutover .

---

In recent years , Yun He AI The rise of has collided with many new technologies and ideas in the field of database . I want to know how databases will be innovated and innovated in the cloud era ？ How to move and build the core database of financial industry safely and smoothly ？ Come on 2020 DAMS China data intelligent management summit Shanghai station To find the answer ：

• 《All in Cloud Time , Next generation cloud native database technology and trend 》 Vice president of Alibaba Group / Chief database scientist of Dharma Hall Fei Fei Li （ Flying cutter ）
• 《 Demand and selection of distributed database in financial industry 》 Industrial and Commercial Bank of China Deputy general manager of the third Department of data center system Zhao Yongtian
• 《 From the evolution of self research, we can see the distributed database 》 China UnionPay Cloud Computing Center team leader Zhou Jiajing
• 《 Open source database MySQL Application practice in Minsheng Bank 》 Minsheng Bank project manager Xu Chunyang