Baidu proposes a dynamic self distillation method to realize dense paragraph retrieval by combining interactive model and double tower model

Thesis title ：

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

Thesis link ：

http://arxiv.org/abs/2205.09153

This paper proposes two distillation strategies for dense retrieval models ： Interactive distillation (Interaction Disitllation) And cascade distillation (Cascade Distillation), If these two distillation strategies are combined , You get the nouns mentioned in the title of the paper ： Dynamic self distillation (self on-the-fly distillation). Because it is produced by Du factory , The model trained with this distillation strategy is named   ERNIE-Search,ERNIE-Search stay MARCO and NQ Retrieval data sets have been obtained SOTA performance （ Although it was surpassed within a few days ）.