Record RDS troubleshooting once -- RDS capacity increases dramatically

Customer afternoon 7 Click to report the failure ,

4 A.m. RDS The capacity suddenly changed from 500G Upgrade to 900G, Take up all the space , The business cannot run normally

The analysis reason , see

1、MySQL The solution to the problem that the instance disk space is full due to temporary files

Question why

MySQL The instance may be due to the sorting of query statements 、 grouping 、 Temporary table file generated by associated table , Or before a big transaction is committed binlog cache file , The instance space is full , To avoid data loss ,RDS The instance will be automatically locked , After disk lock , Will not be able to write .

Solution

After upgrading the instance storage space, you can unlock the instance

show processlist

You can view the process processing , exclude time The delay is relatively large , Has not been resolved , It's probably the problem

2、MySQL Solution to the problem that the instance disk space is full due to system files

Question why

MySQL Instances may be caused by queries that do not end for a long time ibdata1 The file is too large to shrink , The instance space is full , To avoid data loss ,RDS The instance will be automatically locked , After disk lock , Will not be able to write .

After upgrading the instance storage space, you can unlock the instance

 

matters needing attention

• There is a delay in cleaning up temporary files , Please wait patiently for the space used by the instance to decline .

• because MySQL 5.7 Start using independent temporary table spaces ibtmp1, You can free up space by restarting the instance . about MySQL5.5/5.6 example , Without upgrading disk space , A better solution is to buy the same configuration in the same region and the same availability zone RDS example , adopt DTS Tools migrate data to new instances .

3、MySQL Binlog The solution to the problem that the file causes the instance space to be full

Question why

MySQL Instances may be generated quickly due to large transactions Binlog file , The instance space is full , To avoid data loss ,RDS The instance will be automatically locked , After disk lock , Will not be able to write .

matters needing attention

• Binlog The file records the transaction information of the instance , yes MySQL Instance high availability 、 The foundation of recoverability , It is recommended not to close . You can upload with one click Binlog To Alibaba cloud OSS To free up disk space or modify Local Binlog Set up .

• clear Binlog File is delayed , Please wait patiently for the space used by the instance to decline .
• One click upload Binlog The cleanup task will be submitted asynchronously in the background , The cleanup task will delete the written Binlog Upload to OSS（ Not purchased by users OSS） On , Then delete from the instance space Binlog file , Currently being written Binlog The file was not written , It can't be cleaned up . therefore , The cleaning process will be delayed , It is recommended that you click One click upload Binlog Then wait patiently for a certain time , Do not click this button more than once , You can check whether the used space is reduced in the basic information page .
• because DML Wait for the operation （ For example, those involving large fields DML operation ） Lead to rapid generation Binlog, Uploading may occur Binlog The processing speed of transferring files to the backup space and deleting them from the instance space cannot keep up with the instance generation Binlog The speed of the file , under these circumstances , It is recommended to consider upgrading disk space , And check Binlog Reasons for rapid generation .

4、ySQL Data file causes instance space full solution

Question why

MySQL The instance may be full because the data file has not been collated for a long time , To avoid data loss ,RDS The instance will be automatically locked , After disk lock , Will not be able to write .

matters needing attention

• Please make sure there is data backup before deleting the table , So as not to cause loss .

• Recommended drop or truncate Command to free space .
• optimize Operation will lock the table , It is recommended to operate in the low peak period .
• There is a delay in cleaning up data files , Please wait patiently for the space used by the instance to decline .

Look at the analysis data

SELECT file_name, concat(TOTAL_EXTENTS,'M') as 'FIle_size' FROM INFORMATION_SCHEMA.FILES order by TOTAL_EXTENTS DESC

After analyzing the cause of the fault, it is 4 When the customer orders FALSE SQL Statements cause congestion

Resume after restart