Reading the thesis "sentence embeddings using Siamese Bert networks"

Article address ：https://arxiv.org/abs/1908.10084

Article content

  BERT and RoBERTa Good results have been achieved in some tasks , But when dealing with tasks related to sentence pairs , They require that 2 Input three sentences together into the network for training , And this will lead to high computational costs . Therefore, the author of this paper proposed Sentence-BERT(SBERT), It makes twin or triple network architecture （siamese and triplet network） Generate meaningful sentences embedded , Then, the cosine similarity can be used for comparison , Because the above network architecture can encode sentences in parallel , Therefore, the processing time of sentences to tasks can be greatly reduced .

Article model

  SBERT stay BERT A pooling operation is added to the model , In fact, the author has tried three ways , Namely ：[CLS]、MEAN-strategy、MAX-strategy, The best way to pass the experiment is MEAN-strategy.
   The author has set 3 Structure and objective function .
  Classification Objective Function： Classification objective function

   The classification structure is shown above , In order to better learn 2 Interaction between sentences , The following strategies are used ：

Loss is the common cross loss entropy . Of course, the author also tested several strategies of interaction or feature fusion , The results are as follows ：

  Regression Objective Function： Regression objective function ：

   This is a simple calculation 2 Cosine similarity between sentences , In fact, the author has also tried European distance or Manhattan distance , But cosine similarity is the best . Use the mean square error as the loss function .
  Triplet Objective Function.： Sansheng network ： This is the given anchor sentence a, Example p, Negative example sentence n, The training method is to let a and p As far as possible , Give Way a and n As far as possible , The formula is as follows , The author here simply uses Euclidean distance .

Article summary

   For experiments and training （ Wikipedia 、 Sentence pair classification task ） No more details about , I feel that the most valuable part of this article is the interactivity of this sentence , How to classify the network , And the cosine similarity of the regression network 、 Various distances, etc , Maybe it can be used in other tasks , And through 2 The same encoder can encode sentences in parallel, which can be fully hardware , Saving time .