Problems caused by List getting the difference

Background

When saving data in batches, it needs to be compared with the database.If the current data exists, it will be removed from the List collection.

If the List is deleted in batches, removeAll is generally considered.

The operation process is as follows:

1. Match the external unique field of the existing batch data with the data in the database, and return the existing List object

2. Use the removeAll function to delete in batches to obtain the current difference.

After using the removeAll function, data duplication is directly caused when saving in batches.

Api description

Operation Type	Method	Description
Intersection	listA.retainAll(listB)	After calling the method, ListA becomes the intersection of the two sets, and ListB remains unchanged
Difference	listA.removeAll(listB)	After calling the method, ListA becomes the difference of the two sets, and ListB remains unchanged
Union	1.listA.removeAll(listB) 2.listA.addAll(listB)	To remove duplicates, first take the difference and then the union.ListA becomes the union of the two sets, ListB remains the same

removeAll source code

Saying anything is false, look at the removeAll source code brand.

public boolean removeAll(Collection c) ​​{return batchRemove(c, false, 0, size);}...boolean batchRemove(Collection c, boolean complement,final int from, final int end) {Objects.requireNonNull(c);final Object[] es = elementData;int r;// Optimize for initial run of survivorsfor (r = from;; r++) {if (r == end)return false;if (c.contains(es[r]) != complement)break;}int w = r++;try {for (Object e; r < end; r++)if (c.contains(e = es[r]) == complement)es[w++] = e;} catch (Throwable ex) {// Preserve behavioral compatibility with AbstractCollection,// even if c.contains() throws.System.arraycopy(es, r, es, w, end - r);w += end - r;throw ex;} finally {modCount += end - w;shiftTailOverGap(es, w, end);}return true;}

As we can see, each object needs to be compared in a loop.

 for (r = from;; r++) {if (r == end)return false;if (c.contains(es[r]) != complement)break;}

The data found from the database contains other fields such as ID.In this way, the properties of the two objects are different.So it will return false.This defeats the purpose of weight loss.

Custom objects can be deduplicated using the Stream method of JDK8+ below.

Match the data already in the library (exitList) with the data that needs to be saved (entityList), and pick out the data that does not exist in the library and put it into a new List (saveDataEntityList).

The pseudo code is as follows:

saveDataEntityList = entityList.stream().filter(f -> !exitList.stream().map(Entity::getId).collect(Collectors.toList()).contains(f.getId())).collect(Collectors.toList());

After testing, found OK, no duplicate data.

Summary

removeAll is suitable for subset exact matching and basic type operations. It is recommended that when customizing objects, do not use the removeAll method, but use the stream method.