Fault 007: The dexp derivative is inexplicably interrupted

DM技术交流QQ群：940124259

1. 问题描述

A unit of party b to implement personnel,Recently has beendexpExport data in the process of repeated“Code:-70037 字符串不完整”Confused by mistake,Every time the two export always fail,Last two months can't find the wrong key points and breakthrough.按常理来说,Export data about how easy it is,恰好相反,The more often the simple things more let you doubt life,Is the prophase work careless to today's end.

好,I am no longer hold in suspense！The cause of this guide is not out of data,其一dexpIn the process of export data to check the data from,Lack of character break code,Will mercilessly interrupt export,Even a negotiable didn't have to（Please don't ridicule,微笑面对人生^_^）;Secondly some rows in the table data is illegal characters,The character stil,Before induced by,Are they originally fromMySQL迁移数据时,Not strict inspection data,Many string is truncated.

Told in advance the cause of the problem,It feels a bit like let everyone read it,To know the end.Could you carefully read my hard work,Truly understand the problem of,To master the thinking of problem analysis and problem solving.谦虚使人进步,切记！！！The scene seen too many thinksDBA,或开发人员,Always feel very cowB.A.I,And actually when faced with,挠脑袋,Learn so much method,看了这么多书,Why don't use on.

上菜,截图
He tried to right management tools export,Try againdexp命令行导出,Both thrown fault interrupt follow-up export operation.

---

2. 见招拆招

2.1 Line sampling precision positioning error

Try to narrow the scope of the export data error,Picking behavior characteristic obvious error the first line of the（Trigger an error line position for the first time）.Retrieves the position of the this line,The next screen easier.
思路：

1. Comprehensive analysis of the data content more features in the table,Such as grouped by date/time constant constant values in here,观察数据分布,In turn drawn from small amount of data and abnormal content record line.
2. Delimits the scope of data rows exported step by step,Until removed from the location of the data for the first time interrupt export,So in management tools by paging query way,Location query the corresponding row.

-- 1. Through his own experience,Choose has the characteristics of the grouping columns,First of all to understand the data in the table distribution（此处截图省略,Understand the core idea can be）
select to_char(trunc(ds_create_time)), count(*) 
from "YJSCP"."graduate_recruit_enroll_resume"
group by to_char(trunc(ds_create_time))
order by 1 desc;


-- 2. single2022-03-31那天的数据（4百多条）导出测试
-- rownum<=100 rownum<=80 rownum<=70 rownum<=73 Gradually close the truth（Of course also can write a script to get,This error does not have many table,So there is no need to waste time to write the script automatically catch out）
./dexp "YJSCP"/'"xxxxxxxx"'@LOCALHOST:5237 DIRECTORY=/tmp \
   FILE=res5.dmp feedback=1 \
   TABLES="\"YJSCP\""."\"graduate_recruit_enroll_resume\"" \
   QUERY="WHERE to_char(trunc(ds_create_time))='2022-03-31' and rownum<=80" LOG=exp_2022_07_27_10_58_09.log \
   LOG_WRITE=N

-- Had been positioning to73-74行,To enter the next step the content of the detailed analysis

---

2.2 大胆推测、与人交流

Observe a an error line data content characteristics,Look for character type field,A special check for the special characters（XML格式/HTML格式/类似C编程语言）,Belong to the first watch.
Communicate and understand the implementation of the business table design,Understanding of the meaning of some fields,If there is one field notes,另当别论,Or a field name is easy to read,Which fields can be suggested, may exist abnormal data content.
Why emphasize trying to understand the meaning of each field in the table？If a table more than one hundred fields,A lot of text type column is the character,Artificial screening slowly,时间太长,效率极低.In a short period of time, quickly trying to locate certain fields,Wouldn't it be cool.
So says the technology to do,Don't study hard work（In some cases may be understood as mulling over）,Also learn to communication between people,技术交流.

-- 1. Through the paging query positioning error to the range
select top 73,2
from "YJSCP"."graduate_recruit_enroll_resume"
where to_char(trunc("ds_create_time")) = '2022-03-31'


-- 2.Focus moved to the character type long field content（此次案例：jcqk）
-- Other content has do glue susceptibility processing.发现jcqkA garbled characters at the end of text content.

-- 3.To further confirm the code of law
-- Ordinary people may want to uselike '%The code symbol',这个是不行的.
-- This forget screenshots,结论是：The code of law of character isEF或EFBFThe characters appear,We should be converted to binary watch.
SELECT *FROM (
SELECT
	"graduate_recruit_enroll_resume_id",
	"ds_create_user_name",
	ds_create_time,
	RIGHTSTR(jcqk, 4) linestr, -- 取后4个字符
	RAWTOHEX (rightstr(jcqk, 1)) bytestr, -- Take the last character of the binary code
	ROWNUM rn  -- 记录行号
FROM
	SYSDBA."graduate_recruit_enroll_resume"
WHERE
	trunc("ds_create_time", 'DD') = date'2022-03-31'
 ) WHERE rn IN (74, 311, 381); -- Take an error line,Observe a character encoding rule
 
-- 4. Check with the code word in the text character
/** Strict operation,Should be to continue to check whether there is in the middle of the text stil. 事实上,当时场景,Not to the follow-up to proceed,For the above at the end of the code has to do correction after three problems,再次dexp导出2022-03-31The data is no longer an error,问题基本得到了解决. 特别提示：The best backup the original table,Do the test with the table backup correction.Don't take it for granted in the production of library business messing around on the table. **/

Give friends more confident to distinguish,关于EF BFUnder the code corresponding to the various character set of characters to describe.
参考网址： https://www.haomeili.net/Code/DeCoding?bytes=EFBF&isHex=True

2.3 DTSMigration backup the original table

No matter how you use the backup table can be,目的是保证数据的安全性.Another purpose of I,通过DTSTools to migrate a table as a backup table,看DTSTool can automatically get rid of the code of the character,Province behind the manual check and repair the data.At that time out this idea,Little work and well done,何乐而不为？It's a pity that validation by himself,DTSMigration backup plan won't work,Had to find out the problem by hand data row,Fix the code field.
The second purpose although I did not achieve the desired results,But the first goal must be,保护原数据,Test the repair backup table data.
When moving the table backup toSYSDBA用户模式下,A full table of location and tracking,Line selection problem.

SELECT *FROM (
SELECT
	"graduate_recruit_enroll_resume_id",
	"ds_create_user_name",
	ds_create_time,
	RIGHTSTR(jcqk, 4) linestr,
	RAWTOHEX (rightstr(jcqk, 1)) bytestr,
	ROWNUM rn  -- The line number more accurate,Because did not go any output index scan,是以rowidFocus in the form of table.（Yo buddy details to consider,Write eachSQLMust conform to the scene at that time）
FROM
	SYSDBA."graduate_recruit_enroll_resume"
WHERE
-- trunc("ds_create_time", 'DD') = date'2022-03-14'
 ) 
 WHERE bytestr IN ( 'EF', 'EFBF');

DBvear客户端管理工具,Retrieval problem data line screenshot

2.4 Understand the logic of the mistakes（可选）

Verify export,Each error output is related to row,That there are a few lines of abnormal data,How many times we print the same mistake.（可选内容）
I like to study more through it,Add one more section of data error validation and guess.
举例：拿2022-03-31Export data to observe a day,An error three characters incomplete,Meet my expectations.After the conclusion of you deal with a similar error message,Have a information about control.

./dexp "YJSCP"/'"xxxxxxxx"'@LOCALHOST:5237 DIRECTORY=/tmp \
   FILE=res5.dmp feedback=1 \
   TABLES="\"YJSCP\""."\"graduate_recruit_enroll_resume\"" \
   QUERY="WHERE to_char(trunc(ds_create_time))='2022-03-31'" LOG=exp_2022_07_27_10_58_09.log \
   LOG_WRITE=N

2.5 Repair data to verify export

Now that meet at first how many times an error will be how many lines of data error inference problem,2.4Section query results of34In character stil,After comparing before party b's export in the log,Is in conformity with the conclusion,Then try to fix the problem line data.
After repair again trying to export the whole form,如果不再报错,Has completed our goal the repair,Otherwise continue to dig in the middle of the cipher text.

-- 1. Fix the code line
update SYSDBA."graduate_recruit_enroll_resume" a
SET jcqk = leftstr(jcqk, LENGTH(jcqk) - 1)
WHERE RAWTOHEX (rightstr(jcqk, 1)) IN ('EF', 'EFBF');

COMMIT;


-- 2. Trying to export the whole form
./dexp "SYSDBA"/'"xxxxxx"'@LOCALHOST:5237 DIRECTORY=/tmp  \
    FILE=res5.dmp feedback=1 TABLES="\"SYSDBA\""."\"graduate_recruit_enroll_resume\""  \
     LOG=exp_2022_07_27_10_58_09.log LOG_WRITE=N

2.6 In the production of library data repair mission

All data to repair the entire front operation finished,As for the production of library business table on this matter,Have to let them to coordinate a time to do data recovery .
Methods and steps are taught them,千叮咛万嘱咐,A must to make a backup operation again.
备份：Can the whole database backup,Can form physical backup,A copy of can watch,可以DTSBackup migration to other mode, and so on.....

---

3. 心得分享