当前位置:网站首页>PT OSC deadlock analysis
PT OSC deadlock analysis
2022-07-06 12:12:00 【wx5caecf2ed0645】
1、 Before going online one day , Take a look at the online SQL list , Found a SQL Large table ( About 30 million records ) Add a field , This table 24 There will be business use every hour , Only in the evening, the operation is not so frequent . In order to reduce the impact on the business , Decide to use pt-online-schema-change Tool to update the table structure .
pt-online-schema-change: It is online for big tables alter operation , And try to avoid affecting online business , This is the best mysql One of management work , It is often used in operation and maintenance .
- 1.
(1) The environment of occurrence :
- 1.
-
percona-toolkit :3.0.4
-
mysql:5.6.27-log
-
isolation:Read committed
(2) Problem description :
pt-online-schema-change After the tool finishes changing the table structure , The next day, the developer came over and said to help check the online database around 11 o'clock last night , From the perspective of application error reporting , About half an hour before and after this point, the application reported threeorfour deadlocks , The business side began to swear .
- 1.
(3)error log
After hearing what development said , Right now mysql error( It is recommended that each deadlock be recorded error log) Analyze , The deadlock contents are as follows :
*** (1) TRANSACTION:
TRANSACTION 164993957148, ACTIVE 3 sec setting auto-inc lock
mysql tables in use 2, locked 2
LOCK WAIT 6 lock struct(s), heap size 1184, 3 row lock(s)
MySQL thread id 189071372, OS thread handle 0x7f9e5e82d700, query id 310630851540 10.10.168.169 prodect_user update
REPLACE INTO `pppppp_ccc`.`_t_good_price_new` (`gid`,`price`,`shop_id`,`updatetime`,...) VALUES (`NEW.gid`,`NEW.price`,`NEW.shop_id`,`NEW.updatetime`,...)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `pppppp_ccc`.`_t_good_price_new` trx id 164993957148 lock mode AUTO-INC waiting
*** (2) TRANSACTION:
TRANSACTION 164993956744, ACTIVE 3 sec fetching rows
mysql tables in use 2, locked 2
53 lock struct(s), heap size 6544, 4024 row lock(s), undo log entries 3481
MySQL thread id 189078502, OS thread handle 0x7f9e539e7700, query id 310630850764 10-10-8-8 10.10.168.168 pt_user Sending data
INSERT LOW_PRIORITY IGNORE INTO `pppppp_ccc`.`_t_good_price_new` (`gid`,`price`,`shop_id`,`updatetime`,...) SELECT `gid`,`price`,`shop_id`,`updatetime`,... FROM `pppppp_ccc`.`t_good_price` FORCE INDEX(`PRIMARY`) WHERE ((`gid` >= '32081571')) AND ((`gid` <= '32087240')) LOCK IN SHARE MODE /*pt-online-schema-change 19391 copy nibble*/
*** (2) HOLDS THE LOCK(S):
TABLE LOCK table `pppppp_ccc`.`_t_good_price_new` trx id 164993956744 lock mode AUTO-INC
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 2905 page no 349991 n bits 224 index `PRIMARY` of table `pppppp_ccc`.`t_good_price` trx id 164993956744 lock mode S locks rec but not gap waiting
Record lock, heap no 109 PHYSICAL RECORD: n_fields 22; compact format; info bits 0
*** WE ROLL BACK TRANSACTION (1)
2017-10-23 23:06:27 7f9e539e7700InnoDB: transactions deadlock detected, dumping detailed information.
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
2、 analysis
According to the deadlock information, we can get 2 Messages :
- 1.
-
(1) Transaction one is waiting "_t_good_price_new" Tabular AUTO-INC Table locks ;
-
(2) Business 2 To obtain the "_t_good_price_new" Of AUTO-INC Table locks , wait for "t_good_price" The record lock of .
MySQL The deadlock information record of is not detailed , Just by the two above SQL, It won't cause deadlock at all , Then we can only find out what the whole transaction is SQL Composed of , And the order .
Business 1 Of replace into sentence , It's obviously running pt-osc Generated by the trigger created , When the records of the original table are updated , Trigger and record with replace Mode to synchronize to the new table , So is there any deadlock in this situation ? The answer is yes .
- 1.
-
Business 1:
-
(1)t_good_price Update according to conditions , Yes t_good_price Hold exclusive RECORD LOCKS;
-
(2) The trigger is triggered after the update , And then to replace The way to insert _t_good_price_new surface , Need to be right _t_good_price_new Hold an implicit self incrementing lock .
- 1.
-
Business 2:
-
(1) We can know from the last article ,insert into select from Lock sequence of , Business 2 First of all, _t_good_price_new Add the table level self incrementing lock ;
-
(2) After locking the new watch , Then according to the primary key in the condition id To apply for the original form t_good_price The record lock of .
From above , Due to transaction 1 Update the original table first t_good_price, Put an exclusive lock on the updated records , When the trigger has not been triggered , Business 2 Start execution , This is the time for business 2 Now lock the new watch , When it applies for a record level shared lock on the original table , It was found that some records were locked exclusively , So we need to wait . When the transaction 1 Trigger triggered , You need to obtain a self incrementing lock for the new table , Cause a loop , Generate deadlock .
- 1.
-
Business 1:
-
hold :t_good_price On the table record x lock
-
wait for :_t_good_price_new On the table auto-inc lock
-
Business 2:
-
hold :_t_good_price_new On the table auto-inc lock
-
wait for :t_good_price The record on the table S lock
The above conditions are : Business 1 The updated record happens to be a transaction 2 Need to add S The lock , This becomes a perfect deadlock .
Combined with our business , At that time, many commodity prices needed to be revised , It is also a batch operation , So the above things happened , Interested , You can continue to do the following tests to reproduce .
3、 repeat
(1) Preparation conditions :
- 1.
(1) Build table
create table test_deadlock (id int auto_increment primary key,name varchar(10));
(2) insert data
insert into test_deadlock select null,' Zhang San ';
insert into test_deadlock select null,' Li Si ';
insert into test_deadlock select null,' Wang Wu ';
insert into test_deadlock select null,' Chen Liu ';
insert into test_deadlock select null,' Panax notoginseng ';
(3) Build a new watch , simulation pt
create table _test_deadlock_new like test_deadlock;
alter table _test_deadlock_new add column age int default null;
(4) Build triggers , Only newer ones are built here , simulation pt, Because there are few records , Update soon , So when the trigger triggers sleep 5s, Increase the operational time
delimiter //
create trigger pt_osc_test_test_deadlock_upd after update on test.test_deadlock for each row begin declare x int; set x = sleep(5); delete IGNORE from test._test_deadlock_new where !(OLD.`id` <=> NEW.`id`) and test._test_deadlock_new.id <=> OLD.`id`;replace into test._test_deadlock_new (`id`, `name`) values (NEW.`id`, NEW.`name`);END//
delimiter ;
(5) The data in the table
mysql> select * from test_deadlock;
+----+--------+
| id | name |
+----+--------+
| 1 | Zhang San |
| 2 | Li Si |
| 3 | Wang Wu |
| 4 | Chen Liu |
| 5 | Panax notoginseng |
+----+--------+
(2) Operation steps :
session 1:
begin;
update test_deadlock set name = ' Dried tangerine or orange peel ' where id = 4;
- 1.
- 2.
session 2( Be careful , Need to be in 5s Internal operation ):
begin;
insert into _test_deadlock_new(id,name) select id,name from test_deadlock where id > 2 and id <5 lock in share mode;
- 1.
- 2.
5s As soon as the clock arrives ,session 1 It's a mistake , The information is as follows :
mysql> update test_deadlock set name = ' Dried tangerine or orange peel ' where id = 4;
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
- 1.
- 2.
The lock information is as follows :
*** (1) TRANSACTION:
TRANSACTION 1880306, ACTIVE 1 sec fetching rows
mysql tables in use 2, locked 2
LOCK WAIT 5 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1
MySQL thread id 3342, OS thread handle 0x7f209e5e0700, query id 3056466 localhost root Sending data
insert into _test_deadlock_new(id,name) select id,name from test_deadlock where id > 2 and id <5 lock in share mode
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 1642 page no 3 n bits 72 index `PRIMARY` of table `test`.`test_deadlock` trx id 1880306 lock mode S locks rec but not gap waiting
Record lock, heap no 5 PHYSICAL RECORD: n_fields 4; compact format; info bits 0
0: len 4; hex 80000004; asc ;;
1: len 6; hex 0000001cb0f1; asc ;;
2: len 7; hex 120000271805ca; asc ' ;;
3: len 6; hex e99988e79aae; asc ;;
*** (2) TRANSACTION:
TRANSACTION 1880305, ACTIVE 5 sec setting auto-inc lock, thread declared inside InnoDB 5000
mysql tables in use 2, locked 2
4 lock struct(s), heap size 1184, 1 row lock(s), undo log entries 2
MySQL thread id 3341, OS thread handle 0x7f209f59e700, query id 3056468 localhost root update
replace into test._test_deadlock_new (`id`, `name`) values (NEW.`id`, NEW.`name`)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 1642 page no 3 n bits 72 index `PRIMARY` of table `test`.`test_deadlock` trx id 1880305 lock_mode X locks rec but not gap
Record lock, heap no 5 PHYSICAL RECORD: n_fields 4; compact format; info bits 0
0: len 4; hex 80000004; asc ;;
1: len 6; hex 0000001cb0f1; asc ;;
2: len 7; hex 120000271805ca; asc ' ;;
3: len 6; hex e99988e79aae; asc ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
TABLE LOCK table `test`.`_test_deadlock_new` trx id 1880305 lock mode AUTO-INC waiting
*** WE ROLL BACK TRANSACTION (2)
------------
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
Those who are interested can study , So execute pt-osc There are also risks on busy servers . For example, we encounter this situation , just insert The record to the new table is also the data updated by the business , Although the probability is very low , But there will also be such a situation . It can also be used through --chunk-size And some other parameters , Try to reduce the probability of this happening .
边栏推荐
- Basic knowledge of lithium battery
- Navigator object (determine browser type)
- Dependency in dependencymanagement cannot be downloaded and red is reported
- RT-Thread 线程的时间片轮询调度
- Bubble sort [C language]
- C语言回调函数【C语言】
- GCC compilation options
- Pytorch-温度预测
- Programmers can make mistakes. Basic pointers and arrays of C language
- Pat 1097 duplication on a linked list (25 points)
猜你喜欢
Esp8266 uses Arduino to connect Alibaba cloud Internet of things
Basic operations of databases and tables ----- creating data tables
[esp32 learning-2] esp32 address mapping
Implementation scheme of distributed transaction
Gallery's image browsing and component learning
Custom view puzzle getcolor r.color The color obtained by colorprimary is incorrect
JS regular expression basic knowledge learning
RT-Thread的main线程“卡死”的一种可能原因及解决方案
Cannot change version of project facet Dynamic Web Module to 2.3.
Fashion-Gen: The Generative Fashion Dataset and Challenge 论文解读&数据集介绍
随机推荐
Gallery's image browsing and component learning
Selective sorting and bubble sorting [C language]
inline详细讲解【C语言】
Types de variables JS et transformations de type communes
FreeRTOS 任务函数里面的死循环
sklearn之feature_extraction.text.CountVectorizer / TfidVectorizer
AMBA、AHB、APB、AXI的理解
E-commerce data analysis -- User Behavior Analysis
荣耀Magic 3Pro 充电架构分析
Variable parameter principle of C language function: VA_ start、va_ Arg and VA_ end
优先级反转与死锁
【ESP32学习-1】Arduino ESP32开发环境搭建
Basic use of pytest
程序员老鸟都会搞错的问题 C语言基础 指针和数组
JS object and event learning notes
Characteristics, task status and startup of UCOS III
高通&MTK&麒麟 手機平臺USB3.0方案對比
Arduino JSON data information parsing
高通&MTK&麒麟 手机平台USB3.0方案对比
[template] KMP string matching