MySQL finds duplicate data based on one or more fields

Previously, a function similar to wechat or Alipay billing was launched , There is a statistical table . According to the users ID、 year 、 The order quantity and total order amount of the current month can be found by monthly query . When creating the index , It's based on the user ID、 year 、 The union index created by the month field .

Table creation SQL

CREATE TABLE `stat_order_count` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'id',
  `user_id` bigint NOT NULL DEFAULT '0' COMMENT ' user id',
  `year` int NOT NULL DEFAULT '0' COMMENT ' year ',
  `month` int NOT NULL DEFAULT '0' COMMENT ' month ',
  `order_count` int NOT NULL DEFAULT '0' COMMENT ' Number of times recorded ',
  `order_amount` decimal(32,4) NOT NULL DEFAULT '0.0000' COMMENT ' Work recording amount ',
  `record_date` date NOT NULL DEFAULT '1970-01-01' COMMENT ' The recording belongs to the month (YYYY-MM-DD, among DD It has to be for 01)',
  `create_at` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) COMMENT ' Creation time ',
  `update_at` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3) ON UPDATE CURRENT_TIMESTAMP(3) COMMENT ' Update time ',
  PRIMARY KEY (`id`),
  KEY `idx_user_id_record_date` (`user_id`,`record_date`),
  KEY `idx_create_at` (`create_at`),
  KEY `idx_update_at` (`update_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT=' Count the number and amount of orders by month ';

The whole core logic is as follows ：

• If the current user has no data in the current year and month, it will be initialized
• If the current user has data in the current month of the current year, change the statistical information

In the non production environment, there is no master-slave separation in the database , So there is no problem during the test . When the project goes online , The database of the production environment is separated from the master and slave . The main database is used to add data , The data query operation is from the database . If the interval between two pieces of data is short , The first data is added to the main database , The second data query cannot be found from the database . The same user in the current year , There will be more than one in the current month . In this way, multiple items will be found when querying the user's monthly bill information , You're going to report a mistake .

At this time, it is necessary to query the duplicate data for deletion , Then delete the original union index , Add unique index . The following is how to query the duplicate data SQL：

Query for duplicate data .sql

select b.id
  from(
select id
  from stat_order_count a
 where(a.user_id, a.year, a.month)  in(
select user_id, year, month
  from stat_order_count
 group by user_id, year, month
having count(*)> 1))  b
 where b.id not in(
select min(id)  as id
  from stat_order_count
 group by user_id, year, month
having count(*)> 1)

According to the inquiry ID Delete data , Then update the index .

Update index .sql

>  Delete the original union index 
alter table stat_order_count drop index idx_user_id_year_month;

>  Add unique index 
alter table stat_order_count add UNIQUE index uk_user_id_year_month(`user_id`,`year`,`month`);

Finally, the Alibaba cloud primary database is forced to be queried during data query ：

Forced query of main database .sql

/*FORCE_MASTER*/ SELECT * FROM stat_order_count 
where user_id = ? and year = ? and month = ?;

Reference article ：

• https://www.jb51.net/article/152155.htm
• https://help.aliyun.com/knowledge_detail/52221.html