Explain the three ways to remove duplicate data in MySQL

Catalog

One 、 background

Two 、 Three methods of data De duplication are used

1.​ adopt MySQL DISTINCT： duplicate removal （ Filter duplicate data ）

2.group by

3.row_number Window function

3、 ... and 、 summary

One 、 background

Recently, I have been conducting data joint debugging with the system module , One of the requirements is that the relevant data under the two roles ​ After comparison, the latest data will be returned , So I thought of the weight removal , Make a summary again .

Two 、 Three methods of data De duplication are used

1.​ adopt MySQL DISTINCT： duplicate removal （ Filter duplicate data ）

​ 1.1. In the use of mysql SELECT Statement returns all matching rows when querying data .

SELECT t.age FROM t_user t

You can see that the query result returns 10 Bar record , There are some repetitive age value , Sometimes out of the requirements of data analysis , Record values that need to be de duplicated .

1.2.DISTINCT Key indication MySQL Eliminate duplicate record values .

The grammar format is ：

SELECT DISTINCT < Field name > FROM < Table name >;

SELECT DISTINCT t.age FROM t_user t

  It can be seen from the running results that , The query result only returns 5 Bar record age value , And there are no duplicate values .

ps：

among ,“ Field name ” Name the field to be de duplicated , Multiple fields are separated by commas .

Use DISTINCT You should pay attention to the following points when using keywords ：

DISTINCT Keywords can only be found in SELECT Use in statement .

When de duplicating one or more fields ,DISTINCT Keywords must be at the top of all fields .

If DISTINCT There are multiple fields after the keyword , Multiple fields will be combined to remove duplicate , in other words , Only when multiple fields are combined to be exactly the same will they be de duplicated .

2.group by

SELECT t.age FROM t_user t GROUP BY t.age;3.row_number Window function

  The grammar format is ：

row_number() over (partition by < Field name for grouping > order by < Field name used for sorting within the group >)

De duplication used by the project ：​

select * from (select t.*,row_number() over(partition by t.children_id order by t.update_time DESC) rn from mdm_data_authority_view_info t where t.DATA_CLASS_ID = ' classification id' AND t.DATA_ROLE_ID IN ( ' role id', ' role id' )) where rn = 1; 3、 ... and 、 summary

Here is a detailed explanation of mysql This is the end of the article on the three methods of data De duplication , More about mysql Data De duplication content please search the previous articles of software development network or continue to browse the following related articles. I hope you will support software development network more in the future ！