Why can't Bert completely kill the BM25??

In recent years , Compared with the traditional retrieval model , Large scale pre training transformers The introduction of structure has significantly improved various tasks . This promotion has special model settings on different data sets , However, it is still difficult to fully understand why and how these models can work better .

The ancients said ： Enemy and know yourself , Only in this way can a hundred battles be won . Now the NN The model is not yet a confidant , How to carry out the next upgrade iteration ？ Today, let's take a look at the information retrieval task , be based on Bert Compared with the traditional cross encoder BM25 What are the similarities and differences of sorting algorithms ？

Thesis title ：
How Different are Pre-trained Transformers for Text Ranking?

Thesis link :
https://arxiv.org/abs/2204.07233

And the traditional word - based approach ( Such as BM25 or Query-Likelihood) comparison , Neural information retrieval has recently experienced impressive performance improvements .

Because of BERT Such models have a large number of parameters , So it can deal with sentence structures with long-range dependence and complexity .

When will BERT When applied to sorting , It can be query and doc Build deep interaction between , This allows complex patterns of association to be revealed , Not just simple term matching .

up to now ,BERT The huge performance gain achieved by the cross encoder has not been well explained .

We are right. BERT What kind of features is the model based on to calculate the matching principle of sentence relevance, and the ranking results of the model are consistent with BM25 And other traditional sparse sorting algorithms .

BERT adopt query and doc The interaction between terms directly captures correlation signals , This paper deals with BERT Cross coder (Cross-Encode, Hereinafter referred to as" CE) And BM25 Do some research on the relationship between the sorting algorithm of .

First of all, the following questions are raised ：

RQ1: CE and BM25 What is the difference ?
RQ1.2: CE Whether the BM25 The same results retrieved are better sorted ?
RQ1.3: CE Can be recalled better BM25 Missing results ？

secondly , Quantify separately Exactly match and Soft matching Contribution to the overall effect , Because they constitute the most direct comparison between traditional sparse retrieval and neural retrieval matching paradigm . More specifically , The following questions need to be clarified ：

RQ2: CE Whether it can reflect term perfect match ?
RQ3: CE Can find “ Impossible to be relevant ” The result of ?