RAID5 array recovery case tutorial of a company in Shanghai

Storage used by a company , use RAID5 disk array , The storage suddenly crashes and cannot be started due to unknown reasons ,RAID5 All virtual machines in the array are lost , among 3 Virtual machines are important data , Need to focus on this 3 Virtual machines for data recovery .

One 、 Analyze the storage bottom hierarchy , Develop data recovery plan .

Through the communication with customers and RAID The analysis of array shows that the underlying structure of fault storage is a storage pool composed of several physical disks , Divided into several LUN, Data recovery is required LUN1, It includes the key recovery 3 Virtual machines .

Two 、 Try to reorganize RAID recovery RAID Array data .

When analyzing and reorganizing the array, it is found that... In the user's original storage RAID5 Array common missing 2 Block hard disk , The hot spare has been enabled .（ Restore the fault process to ： After the first hard disk is disconnected, the system starts the hot spare disk for replacement , When the second hard disk is disconnected RAID5 In a degraded state , The third hard disk is disconnected, resulting in RAID Array crash .） In this case, the data of the lost disk cannot be directly obtained through verification , Therefore, you can only use full disks of the same size 0 Reorganize the image （ Due to the dependence on empty images RAID The file system structure will be seriously damaged , It is equivalent to that each strip will be missing two blocks of data , So it is not recommended to do so except in exceptional circumstances ）.

3、 ... and 、 Through reorganization RAID Array extraction LUN.

Through the further analysis of the storage structure, we can get the storage partition MAP block , And then for each LUN The data block pointer of , By writing a data extraction program （ You can also use the North Asia data recovery tool ） extract LUN debris . After extraction, the fragments are spliced , Complete composition LUN.

Four 、 export LUN All virtual machines in the , Try to start .

Because the virtual machine name involves customer privacy and the export process is relatively simple , No more details here . After exporting the virtual machine, try to start , Same as expected , The operating system is damaged and the virtual machine cannot be started .

5、 ... and 、 Extract the files in the virtual machine .

Because the virtual machine could not start , Only files in the virtual machine can be extracted , But most of the files are seriously damaged , Only a few files are available , We have to continue to work out other data recovery schemes .

6、 ... and 、 Extract data by analyzing database pages .

The virtual machine for this data recovery contains mysql database , You can use the particularity of the underlying storage of the database to scan data pages , Extract the data .（ There is an episode in this step , Virtual machine enable snapshot , When both the parent disk and the snapshot file are damaged, the normal merge operation cannot be completed , Use the company's independent research and development VMFS The snapshot merging program performs snapshot merging .） The screenshot of the data recovery process is as follows

7、 ... and 、 obtain mysql Data page and analyze .

according to mysql Data page features scan and export data pages （innodb The engine can use this scheme ,myisam Because no “ Data pages ” The concept is therefore not available ）, Analyze the system table to obtain the information of each user table , According to the... Of each table id Split the data page .

8、 ... and 、 Extract table structure 、 Extract record .

Because the database has been used for a long time , The table structure has also changed many times , In addition, some data is lost after the system table is damaged , There is a lot of resistance in the record extraction process . First, get the table structure of each table in the original version of the database ： The parent disk before the snapshot was merged because it was written earlier , Use the first drop disk for verification to obtain the complete data of this file , Then extract the table structure of each table in the database , After that, the client provided the latest version of the database table creation script .

Two groups of different table structures are used to extract data records and import them into the recovery environment mysql In the database , Then eliminate the garbled data caused by the change of table structure in each table , Finally, the two sets of data are exported as .sql file .

Nine 、 data validation .

Because the database table structures of the two versions are different , So I contacted the application engineer of the client for debugging . Import the platform after debugging , Platform debugging succeeded , This data recovery is successful .