Technology sharing | big transaction blocking show master status

Preface

MySQL The master-slave copy function can build a slave library for MySQL Create an online backup system , But it cannot switch independently ; Third party high availability tools are needed . However, when there are large transactions running, it will block some show sentence ; for example “show master status”, Cause miscarriage of justice .

Scene simulation

1、 Construct 20 million rows of data , Delete as a transaction

2、 New session execution show master status , stay sql The execution of the statement was successful . stay commit Blocked during .

3、show master status be in starting state , That is, the first stage of statement execution , Start-up phase , Prepare resources .

4、 see stack Information ,show master status It's about getting lock_log Blocked when locking

#6  0x0000000000ee8278 in MYSQL_BIN_LOG::get_currrent_log (this=0x1e839c0 <mysql_bin_log>, linfo=0x7f3ea82e62d0, need_lock_log=<optimized out>) at /export/home/pb2/build/sb_0-32013917-1545390211.74/mysql-5.7.25/sql/binlog.cc:5514
......
#6  0x0000000000eef824 in MYSQL_BIN_LOG::change_stage (this=<optimized out>, thd=<optimized out>, stage=<optimized out>, queue=<optimized out>, leave_mutex=<optimized out>, enter_metux=0x1e839c8 <mysql_bin_log+8>) at /export/home/pb2/build/sb_0-32013917-1545390211.74/mysql-5.7.25/sql/binlog.cc:9170

5、 Finally, because the transaction is too big , Final show master status Overtime , Cause failover .

The reason is commit And show master status Blocking and waiting between , Next, analyze the reasons .

Cause analysis ：commit

commit The process is divided into the following stages （ With 5.7.25 For example ）, For atomic operations , Write once .

1、prepare The main function of this stage is to brush redo 、undo ; here binlog It only involves some preparatory actions . reference binlog_prepare function .

static int binlog_prepare(handlerton *hton, THD *thd, bool all)
{
  DBUG_ENTER("binlog_prepare");
  if (!all)
  {
    thd->get_transaction()->store_commit_parent(mysql_bin_log.
      m_dependency_tracker.get_max_committed_timestamp());

  }

2、flush The main function of the stage is persistence redo ; obtain lock_log Locks block other group transactions from writing , And generate flush queue （ The first transaction is leader , This was followed by follower ） And write binlog , At this time, the memory writes but does not brush the disk , reference process_flush_stage_queue function

int
MYSQL_BIN_LOG::process_flush_stage_queue(my_off_t *total_bytes_var,
                                         bool *rotate_var,
                                         THD **out_queue_var)
{
  DBUG_ENTER("MYSQL_BIN_LOG::process_flush_stage_queue");
  #ifndef DBUG_OFF
  // number of flushes per group.
  int no_flushes= 0;
  #endif
  DBUG_ASSERT(total_bytes_var && rotate_var && out_queue_var);
  my_off_t total_bytes= 0;
  int flush_error= 1;
  mysql_mutex_assert_owner(&LOCK_log);

3、sync The stage function is to release lock_log lock , obtain lock_sync Lock blocks other transaction groups from scrubbing , And according to sync_binlog Decide the disk brushing strategy .

if (change_stage(thd, Stage_manager::SYNC_STAGE, wait_queue, &LOCK_log, &LOCK_sync))
  {
    DBUG_PRINT("return", ("Thread ID: %u, commit_error: %d",
                          thd->thread_id(), thd->commit_error));
    DBUG_RETURN(finish_commit(thd));
  }

4、commit Stage release lock_sync lock , obtain lock_commit Lock and wait follower Transaction submission , Release the lock and empty after submitting binlog Cache and judge whether to refresh and archive binlog

Cause analysis ：show master status

show master status Its function is from the last binlog Get in file Executed_Gtid Information and the binlog Executed gtid Information . Its function entry is ： https://github.com/mysql/mysq... 624 Line and get lock_log lock ：

int MYSQL_BIN_LOG::get_current_log(LOG_INFO* linfo, bool need_lock_log/*true*/)
{
  if (need_lock_log)
    mysql_mutex_lock(&LOCK_log);
  int ret = raw_get_current_log(linfo);
  if (need_lock_log)
    mysql_mutex_unlock(&LOCK_log);
  return ret;
}

Conclusion

The example in the case is a single big transaction , Therefore, the group submitted sync Stage , from commit（flush Stage ） as well as show master status You can see in the steps of , The blocking between two operations is caused by lock_log Caused by lock contention , The nature of the lock is lock_log Log lock , Rather than MDL And engine layer lock , If the transaction is larger ,flush Phase write binlog The longer the period and the longer the lock will be held ; Thus blocking other transaction submissions and binlog Of show operation , Therefore, the corresponding solution is ：

1、 Big things can cause io Skyrocketing 、 Problems such as master-slave delay and transaction blocking , stay MGR The environment may even cause replication disruption ,mysql Big things need to be avoided .

2、binlog Is a constantly changing document ,show master status The statement can be changed to select @@global.gtid_executed ; To get GTID Information .