Shell script import and export data

1. Introduce

At work , Many scenes will involve db Import and export of data , In order not to affect normal business , Generally, you will choose to run a batch of scheduled tasks to import and export data when the number of transaction visits at night is small . Our company , Because it is divided into development centers in different regions , There are also many application systems , Therefore, the data dependence between the systems with interconnected businesses , We are Data center Do data transfer , such as : A The system needs to operate the customer information, but it doesn't have this part of information , Customer information is mainly B The system is under maintenance , Now A The system needs B Data of the system .

The plan is B The system publishes customer information and data resources to Data center , because A System to Data center Subscribe to the B Customer information resources of the system , Data center Will forward customer information resources to A The specified directory of the system server , Last A The system only needs to be developed shell The script can import the data in the resource file into the local database .

B System needs A The same is true of the business data of the system , A The system only needs to publish business data resources to Data center ,(A The system development script will db The data is exported as a data file and sent to the data center ), B System subscription A System resources , from Data center Do resource forwarding .

The following is a summary of the work used shell How the script implements db Data import and export , Data center It is an enterprise level data exchange service platform , I won't go into details here .

2. Import data to db2

take MY_CUSTOMER_INFO_[DATE].del File data import MY_CUSTOMER_INFO surface .

3. export db2 data

take MY_CUSTOMER_INFO The data of the table is exported as MY_CUSTOMER_INFO_[DATE].del file .

Exporting data can also use java The program first queries the data , And then through IO Streaming file .

4. Import data to mysql

take MY_CUSTOMER_INFO_[DATE].del File data import MY_CUSTOMER_INFO surface .

Import data section , If null The data wants to be replaced by an empty string , have access to NULLIF Replace after function judgment .

5. export mysql data

take MY_CUSTOMER_INFO The data of the table is exported as MY_CUSTOMER_INFO_[DATE].del file .

6. Java Program call shell Script

Use java.lang.Runtime Class implementation shell Call and execute the script .

7. Problems encountered

(1) above Java The program calls and executes shell Script , Note that the server system type is Linux still AIX System , Otherwise, the incoming parameters cannot be received .

(2) windows Environmental Science (\r\n) and linux Environmental Science (\r) The line breaks are different , Lead to in windows Edited by the system shell Script in Linux An error is reported on the server , Need to use vim The editor will \r\n Replace with \r.

Script formatting , \r\n Replace with \r.

:set ff=unix

Script formatting , \r Replace with \r\n.

:set ff=dos