What if the self incrementing IDs of online MySQL are exhausted?

on-line MySQL Self increasing of id What about exhaustion ？

MySQL Self increasing of id They all define initial values , And then step by step . Although there is no upper limit for natural numbers , But it defines the length of bytes that represent this number , Computer storage has an upper limit . such as , Unsigned integer (unsigned int) yes 4 Bytes , The upper limit is 2^32 - 1. That's growing id run out , What will happen? ？

Table definition self increment id

The logic of the table definition after the increment reaches the upper limit is ： Apply for the next one id when , The resulting value remains unchanged .

mysql> create table t(id int unsigned auto_increment primary key) auto_increment=4294967295;
Query OK, 0 rows affected (0.01 sec)

mysql> insert into t values(null);
Query OK, 1 row affected (0.00 sec)

mysql> show create table t;
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                                      |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| t     | CREATE TABLE `t` (
  `id` int unsigned NOT NULL AUTO_INCREMENT,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=4294967295 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

// Successfully inserted a line  4294967295
mysql> insert into t values(null);
ERROR 1062 (23000): Duplicate entry '4294967295' for key 't.PRIMARY'

first insert After success , Of this table AUTO_INCREMENT still 4294967295, Leading to the second insert And get the same self increase id value , Try to execute the insert statement again , Primary key conflict .

2^32 - 1（4294967295） Not a particularly large number , A table with frequent inserts and deletions is likely to run out . When creating a table, you need to consider whether it is possible for your table to reach the upper limit , If you have any , It should be created as 8 Bytes of bigint unsigned.

InnoDB The system is self increasing row_id

If you create InnoDB Table does not specify primary key , be InnoDB Will automatically create an invisible ,6 Bytes of row_id.InnoDB Maintains a global dict_sys->row_id value

All without primary key InnoDB surface , Insert each row of data , Will be the current dict_sys->row_id As the... To insert data row_id, And then put dict_sys->row_id Add 1.

When the code is implemented row_id The length is 8 An unsigned long integer of bytes (bigint unsigned). but InnoDB In design , to row_id All that's left is 6 The length of bytes , In this way, only the last 6 Bytes , therefore row_id Values that can be written to the data table , There are two characteristics ：

1. row_id Write the range of values in the table , It's from 0 To 2^48 - 1
2. When dict_sys.row_id=2^48 when , If you want to apply for inserting data again row_id, Take the last after you get it 6 A byte is 0

Write to the table row_id from 0~2^48 - 1. When the limit is reached , The next value is 0, And then the cycle goes on .

2^48 - 1 It's already big , But if one MySQL Examples live a long time , It's still possible to reach the upper limit .

InnoDB in , succeed in applying for sth. row_id=N after , Write this row of data to the table ; If the table already exists row_id=N The line of , The newly written line will cover the original line .

Verify the conclusion ： adopt gdb Modify the auto increment of the system row_id. use gdb It's for the sake of reproducing the problem , Can only be used in a test environment .

• row_id Used up validation sequence

• row_id Verification of the effect of using up

so , For me gdb take dict_sys.row_id Set to 2^48 after , Insert again a=2 It will appear in the table t The first line of , Because the value of row_id=0.

Then insert a=3, because row_id=1, It covers the previous a=1 The line of , because a=1 In this line row_id It's also 1.

So it should be InnoDB Create auto incrementing primary key in the table ： When the table increases id After reaching the limit , The primary key conflict error will be reported when inserting data again .

After all, overlay data , It means data loss , Affect data reliability ; It's a conflict , Insert the failure , Impact availability . General reliability is better than availability .

Xid

redo log and binlog There's a common field Xid, Used to correspond to transactions .Xid stay MySQL How is the interior generated ？

MySQL A global variable is maintained internally global_query_id

Every time a statement is executed , Assign it to query_id, Then give the variable +1：

If the current statement is the first statement executed by the transaction , be MySQL And at the same time query_id Assigned to the transaction Xid：

and global_query_id Is a pure memory variable , It's cleared after restart . So the same DB example , Different business Xid Could be the same .

but MySQL After restart, a new... Will be regenerated binlog file , This guarantees the same binlog In the file Xid only .

although MySQL Rebooting doesn't result in the same binlog There are two identical Xid, But if global_query_id Reach the upper limit , Will continue from 0 Start counting . In theory, there will still be the same binlog It's the same Xid.

because global_query_id8 byte , ceiling 2^64 - 1. If this happens , Need to meet ：

1. Execute a transaction , hypothesis Xid yes A
2. Next 2^64 Second query statement , Give Way global_query_id go back to A
   2^64 It's too big , This possibility exists only in theory .
3. Start another transaction , This business is Xid It's also A

Innodb trx_id

• Xid from server Layer maintenance

InnoDB For internal use Xid, For connection InnoDB Business and server

but InnoDB Their own trx_id, It's another maintenance business id（transaction id）.

InnoDB One was maintained internally max_trx_id Global variables , Each time you need to apply for a new trx_id when , You get max_trx_id The current value of the , And then max_trx_id Add 1.

InnoDB The core idea of data visibility

Each row of data records the update to it trx_id, When a transaction reads a row of data , Determine if the data is visible , It's through the consistency view of the transaction and this row of data trx_id comparing .

For executing transactions , You can start your information_schema.innodb_trx You can see in the table that trx_id.

Look at the following case ： The transaction trx_id

S2 The execution record of ：

mysql> use information_schema;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select trx_id, trx_mysql_thread_id from innodb_trx;
+-----------------+---------------------+
| trx_id          | trx_mysql_thread_id |
+-----------------+---------------------+
| 421972504382792 |                  70 |
+-----------------+---------------------+
1 row in set (0.00 sec)

mysql> select trx_id, trx_mysql_thread_id from innodb_trx;
+---------+---------------------+
| trx_id  | trx_mysql_thread_id |
+---------+---------------------+
| 1355623 |                  70 |
+---------+---------------------+
1 row in set (0.01 sec)

S2 from innodb_trx The two fields found in the table , Second field trx_mysql_thread_id The thread id. Show threads id, This is to show the thread corresponding to the transaction seen in the two queries id All are 5, namely S1 Thread .

t2 It shows trx_id It's a big number ;t4 All the time trx_id yes 1289, It looks like a normal number . Why is this ？

t1 when ,S1 No updates yet , It's a read-only transaction . For read-only transactions ,InnoDB It doesn't allocate trx_id：

• t1 when ,trx_id The value is 0. And this big number , Just for display
• until S1 stay t3 When the insert,InnoDB To really distribute trx_id. therefore t4 when ,S2 We found that trx_id The value is 1289.

In addition to the obvious modification of class statements , If in select Add... After the statement for update, It's not a read-only transaction .

• update and delete Statement in addition to the transaction itself , It also involves marking old data for deletion , That is to put the data in purge Waiting in the queue for subsequent physical deletion , This operation will also put max_trx_id+1, So add at least 2
• InnoDB The background operation of , For example, such operations as table index information statistics , It will also start internal affairs , So you might see ,trx_id The value is not in accordance with the plus 1 Incremental .

t2 How did the big numbers come from ？

Every time I look up , It is calculated temporarily by the system ：

 Of the current transaction trx The pointer address of a variable is converted to an integer , Plus 248

This ensures ：

• Because the same read-only transaction is during execution , Its pointer address doesn't change , So no matter in innodb_trx Still innodb_locks table , The same read-only transaction trx_id It's going to be the same
• If there are parallel read-only transactions , Of every business trx The pointer address of the variable must be different . such , Different concurrent read-only transactions , Found out trx_id It's different .

Why add 248？

Ensure that read-only transactions are displayed trx_id It's worth more , Under normal circumstances, it will be different from read-write transactions id. but trx_id Follow row_id The logic of is similar to , Defined as 8 Bytes .
Theoretically, it is possible to display a read-write transaction and a read-only transaction trx_id identical . But the probability is very low , There's no real harm , No matter .

Why read only transactions are not allocated trx_id？

• Reduce the size of the active transaction array in the transaction view . Because the currently running read-only transaction , It doesn't affect the visibility judgment of data . therefore , When creating a consistent view of a transaction ,InnoDB You just need to copy the read-write transaction trx_id
• Reduce trx_id The number of applications .InnoDB Perform a normal select sentence , Also corresponding to a read-only transaction . So after optimizing read-only transactions , Ordinary query statements do not need to apply for trx_id, Greatly reduce concurrent transaction requests trx_id Lock conflict

Because read only transactions are not allocated trx_id, obviously trx_id It's slowing down .

but max_trx_id Persistent storage , Rebooting doesn't reset to 0. Theoretically , As long as a MySQL I ran long enough , It's possible max_trx_id achieve 2^48 - 1, And then from 0 Start the cycle .

After reaching this state ,MySQL A dirty read will continue to appear bug：

First of all, put the current max_trx_id Change it to 2^48 - 1. This is repeatable .

• Repeat dirty reading

• Because of the systematic max_trx_id Is set to 2^48 - 1, So in session A Transactions started TA The lowest water level in the world is 2^48 - 1.

t2 when :

• session B Carry out article 1 update Statement transactions id=2^48 - 1
• Second thing id Namely 0 了 , This article update On the data version generated after execution trx_id=0

t3 when ：

session A perform select Visibility judgment of ：c=3 This data version of trx_id（0）, Less than transaction TA The lowest water level in the world （2^48 - 1）, So we think that the data is visible .

But it's dirty reading .

Because the low water level will continue to increase , And the business id from 0 Start counting , Cause the system after that moment , All queries are dirty read .

also MySQL Restart time max_trx_id I don't know 0, The restart MySQL, This bug There is still . What about this bug It only exists in theory ？

Suppose a MySQL Example of TPS yes 50w, Keep doing this ,17.8 That's going to happen in a few years . But from MySQL It's really popular until now , I'm afraid none of them have reached the upper limit yet . however , as long as MySQL Instance service time is long enough , There is bound to be bug.

It can also deepen the understanding of low water levels and data visibility

thread_id

The system holds a global variable thread_id_counter

Every new connection , will thread_id_counter The thread variable assigned to this new connection new_id.

thread_id_counter Defined as 4 Bytes , Therefore, the 2^32 - 1, It will reset to 0, Continue to increase .

But not in show processlist See two identical thread_id. because MySQL Used a unique array

Assign... To a new thread thread_id The logic of time ：

summary

Every kind of self increasing id There are different application scenarios , The performance after reaching the upper limit is also different ：

On the table id When the limit is reached , When you apply again, its value will not change , This leads to a primary key conflict error when you continue to insert data

row_id When the limit is reached , You will return 0 Increase again , If the same row_id, The later data will cover the previous data

Xid Just not in the same binlog Duplicate values appear in the file . Although in theory there will be repeated values , But the probability is very small , Negligible

InnoDB Of max_trx_id Every time MySQL Restart will be saved , So the example of dirty reading mentioned in our article is a necessary one bug, Fortunately, we still have plenty of time

Conclusion

If this article is helpful to you , Or if there's some inspiration , Ask for one key and three links ： give the thumbs-up 、 Comment on 、 Focus on collection , Your support is my biggest motivation for writing .