[server data recovery] case of offline data recovery of two hard disks of IBM server RAID5

Server data recovery environment ：

ibm x3850 Series servers ;
5 Block hard disk composition raid5 disk array ;
linux redhat 5.x operating system ;
oracle database .

Server failure ：

Two hard disks on the server are offline due to unknown failure , Loss of server data , The administrator contacted our data recovery center to recover the data of the failed server . After the server data recovery engineer conducts preliminary inspection on the faulty server , It is found that two hard disks in the server array are offline , The hot spare is not activated . There is no physical failure in the hard disk , No obvious synchronization .

Server data recovery process ：

1、 Shut down the failed server , After marking the fault disk, take out the slot and mount it to the prepared data recovery server environment for image backup . After mirroring the original hard disk, I found that except 2 Plate No 10-20 All other hard disks except the bad sectors are normal . After the image is completed, reinstall the hard disk to the original server .

2、 The server data recovery engineer analyzes the raid structure , Get in array raid Stripe size 、 Check direction 、 Strip rules and meta Area and other information . After analysis, it is found that the best disk sequence structure is 0-1-2-3, defect 3 Number plate , The structure is as follows: ：

 

According to the analysis raid Information virtual build a group raid5 Environmental Science , Data verification after assembly ,200M The above latest compressed packages are decompressed without error , According to this structure, virtual raid Generated on a hard disk , adopt USB Connect the recovered single disk to the original server , Start the fault server through the software and write back the whole disk .

3、 After the data is written back, you cannot enter the operating system , The error message is ：/etc/rc.d/rc.sysinit:Line 1:/sbin/pidof:Permission denied. The data recovery engineer finds the file permissions after restarting the server 、 Time 、 There are obvious errors in size , Analyze and locate the root partition again /sbin/pidof/, The reason for the problem is 2 Bad track of No.1 disk .

4、 Use other disc pairs 2 The damaged area of plate No xor Replenish and recheck the file system , Still wrong , The data recovery engineer is right again inode Table to check , Find out 2 Some nodes in the damaged area of disk No. 1 are shown as ( In the picture 55 55 55 part )：

 

5、 Although described in the node uid It still exists normally , But size 、 attribute 、 The initial allocation block is all wrong . The node information of the original node block is determined through the log and then corrected , again dd Root partition , perform fsck -fn /dev/sda5 testing , The error report is shown in the figure below ：

 

6、 After analysis, we found that , original 3 Disk number is first offline , The intersection of new and old node information leads to multiple nodes sharing data blocks . Data recovery engineers distinguish according to the file to which the node belongs , Clear the wrong node and execute again fsck -fn /dev/sda5, There are still parts located in doc The node under the directory reports an error . Since it does not affect startup, restart the system after forced repair , The system is OK , Starting the database is normal .

7、 The server administrator verifies the server data in person , The verification result indicates that the data is normal , Data recovery successful .