Storage crash MySQL database recovery case

RAID5 disk array , The storage suddenly crashes and cannot be started due to unknown reasons ,RAID5 All virtual machines in the array are lost , among 3 Virtual machines are important data , Need to focus on this 3 Virtual machines for data recovery .

The customer described the fault process as ： After the first hard disk is disconnected, the system starts the hot spare disk for replacement , When the second hard disk is disconnected RAID5 In a degraded state , The third hard disk is disconnected, resulting in RAID Array crash . Let's take a look at how North Asia engineers operate ！

One 、 Analyze the storage bottom hierarchy

Engineers communicate with customers and RAID The analysis of array shows that the underlying structure of fault storage is a storage pool composed of several physical disks , Divided into several LUN, Data recovery is required LUN1, It includes the key recovery 3 Virtual machines .

When the array is analyzed and reorganized, it is found that there are RAID5 Array common missing 2 Block hard disk , The hot spare has been enabled . In this case, the data of the lost disk cannot be directly obtained through verification , Therefore, you can only use full disks of the same size 0 Reorganize the image .

Two 、 Through reorganization RAID Array extraction LUN

Through the further analysis of the storage structure, we can get the storage partition MAP block , And then for each LUN The data block pointer of , By writing a data extraction program 、 extract LUN debris . After extraction, the fragments are spliced , Complete composition LUN.

3、 ... and 、 Extract the files in the virtual machine .

According to the present situation, the Engineer , This failure condition starts after exporting the virtual machine , The virtual machine should fail to start because the operating system is damaged . After exporting the virtual machine, try to start , Same as expected , The operating system is damaged and the virtual machine cannot be started . Because the virtual machine could not start , Only files in the virtual machine can be extracted , But most of the files are seriously damaged , Only a few files are available , We have to continue to work out other data recovery schemes .

Four 、 Extract data by analyzing database pages

The virtual machine for this data recovery contains mysql database , You can use the particularity of the underlying storage of the database to scan data pages , Extract the data .（ There is an episode in this step , Virtual machine enable snapshot , When both the parent disk and the snapshot file are damaged, the normal merge operation cannot be completed , Use the company's independent research and development VMFS The snapshot merging program performs snapshot merging .）

5、 ... and 、 obtain mysql Data page and analyze

according to mysql Data page features scan and export data pages （innodb The engine can use this scheme ,myisam Because no “ Data pages ” The concept is therefore not available ）, Analyze the system table to obtain the information of each user table , According to the... Of each table id Split the data page .

6、 ... and 、 Extract table structure 、 Extract record

Because the database has been used for a long time , The table structure has also changed many times , In addition, some data is lost after the system table is damaged , There is a lot of resistance in the record extraction process . First, get the table structure of each table in the original version of the database ： The parent disk before the snapshot was merged because it was written earlier , Use the first drop disk for verification to obtain the complete data of this file , Then extract the table structure of each table in the database , After that, the client provided the latest version of the database table creation script .

Two groups of different table structures are used to extract data records and import them into the recovery environment mysql In the database , Then eliminate the garbled data caused by the change of table structure in each table , Finally, the two sets of data are exported as .sql file .

7、 ... and 、 data validation

Because the database table structures of the two versions are different , So I contacted the application engineer of the client for debugging . Import the platform after debugging , Platform debugging succeeded , This data recovery is successful .