Text matching - [naacl 2022] GPL

Address of thesis ：https://arxiv.org/abs/2112.07577

《 Text matching ——【EMNLP 2021】TSDAE》 One of the disadvantages of adaptive pre training in is the high computational cost , Because you must first run the pre training on the corpus , Then supervised learning is performed on the labeled training data set . Tagged training data sets can be very large .

GPL（ Unsupervised domain adaptive generation of pseudo tags for dense retrieval ） Overcome the above problems ： It can be applied to fine tune the model . therefore , You can use one of the pre training models and adapt it to a specific domain ：

The longer you train , The better your model is . stay V100-GPU The upper training model is about 1 God .GPL It can be combined with adaptive pre training , To further improve performance .

GPL Work in three stages ：

• query Generate ： For a given text in our domain , We use T5 The model generates possible for a given text query. for example , When your text is “Python is a high-level general-purpose programming language” when , The model may generate something like “What is Python” In this way query. chinese T5 Doc2Query Pre training model address ：https://huggingface.co/doc2query/msmarco-chinese-mt5-base-v1
• Negative example mining ： Next , For build query “What is Python”, We mine negative examples from the corpus passage, I.e query Similar but not relevant to the user passage. Such negative examples passage May be “Java is a high-level, class-based, object-oriented programming language.”.. We use dense retrieval for this kind of mining , That is, we use one of the existing text embedding models to retrieve the given query Correlation passage.
• Pseudo label ： In the negative example mining step , We have retrieved information related to query Actually relevant passage（ Such as “What is Python” Another definition of ）. To overcome this problem , We use Cross-Encoder For all （query、passage） Rate .

Training ： Once we have triples (generated query, positive passage, mined negative passage) And for (query, positive) 、 (query, negative) The score of Cross-Encoder, We can start using MarginMSELoss Training text embedding model ：

The pseudo marking step is very important , With the previous method QGen（《 Text matching ——【NeurIPS 2021】BEIR》） comparison , It improves performance ,QGen take passages Deemed positive （1） Or negative （0）. As we can see in the figure below , For build query (“what is futures conrtact”), Negative example mining steps retrieval and generation query Partially or highly related passages. Use MarginMSELoss and Cross-Encoder, We can identify these passages And teach the text embedding model that these paragraphs are also relevant to a given query .

The following table provides an overview GPL And adaptive pre training （MLM and TSDAE） Comparison . As mentioned earlier ,GPL It can be combined with adaptive pre training ：