当前位置:网站首页>SimCLR:NLP中的对比学习
SimCLR:NLP中的对比学习
2022-07-06 08:53:00 【InfoQ】
论文简介
- 诸神黄昏时代的对比学习
- “军备竞赛”时期的对比学习
- 使用预训练好的 Bert 直接获得句子向量,可以是 CLS 位的向量,也可以是不同 token 向量的平均值。
- Bert-flow^[On the Sentence Embeddings from Pre-trained Language Models],主要是利用流模型校正 Bert 的向量。
- Bert-whitening^[Whitening Sentence Representations for Better Semantics and Faster Retrieval],用预训练 Bert 获得所有句子的向量,得到句子向量矩阵,然后通过一个线性变换把句子向量矩阵变为一个均值 0,协方差矩阵为单位阵的矩阵。
- Sentence-Bert (SBERT)^[Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks],通过 Bert 的孪生网络获得两个句子的向量,进行有监督学习,SBERT 的结构如下图所示。
什么是dropout
dropout和其他数据增强方法进行比较
不同的dropout rate
对比学习评价指标
- alignment 计算所有正样本对之间的距离,如果 alignment 越小,则正样本的向量越接近,对比学习效果越好,计算公式如下:$$
- uniformity 表示所有句子向量分布的均匀程度,越小表示向量分布越均匀,对比学习效果越好,计算公式如下:$$\ell_{\text {uniform }} \triangleq \log \quad \mathbb{E}{x, y \stackrel{i . i . d .}{\sim} p{\text {data }}} e^{-2|f(x)-f(y)|^{2}}$$
无监督
有监督
结果
代码实践
pip install simcse
SimCSE("在这里填写不同版本")
from simcse import SimCSE
model = SimCSE("princeton-nlp/sup-simcse-bert-base-uncased")
embeddings = model.encode("A woman is reading.")
sentences_a = ['A woman is reading.', 'A man is playing a guitar.']
sentences_b = ['He plays guitar.', 'A woman is making a photo.']
similarities = model.similarity(sentences_a, sentences_b)
similarities1 = model.similarity('A woman is reading.', 'A man is playing a guitar.')
similarities2 = model.similarity('He plays guitar.', 'A man is playing a guitar.')
sentences = ['A woman is reading.', 'A man is playing a guitar.']
model.build_index(sentences)
results = model.search("He plays guitar.")
similarities2 = model.similarity('He plays guitar.', 'A man is playing a guitar.')
He plays guitar.
A man is playing a guitar.
边栏推荐
- LeetCode:836. 矩形重叠
- 可变长参数
- LeetCode:236. The nearest common ancestor of binary tree
- Export IEEE document format using latex
- [NVIDIA development board] FAQ (updated from time to time)
- Delay initialization and sealing classes
- Crash problem of Chrome browser
- 【剑指offer】序列化二叉树
- 如何有效地进行自动化测试?
- Using C language to complete a simple calculator (function pointer array and callback function)
猜你喜欢
Chapter 1 :Application of Artificial intelligence in Drug Design:Opportunity and Challenges
TP-LINK 企业路由器 PPTP 配置
Indentation of tabs and spaces when writing programs for sublime text
Variable length parameter
[OC-Foundation框架]---【集合数组】
数学建模2004B题(输电问题)
ESP8266-RTOS物联网开发
Sublime text using ctrl+b to run another program without closing other runs
项目连接数据库遇到的问题及解决
【剑指offer】序列化二叉树
随机推荐
[OC]-<UI入门>--常用控件的学习
如何有效地进行自动化测试?
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
ESP8266-RTOS物联网开发
LeetCode:41. 缺失的第一个正数
vb.net 随窗口改变,缩放控件大小以及保持相对位置
LeetCode:394. 字符串解码
[OC-Foundation框架]---【集合数组】
SAP ui5 date type sap ui. model. type. Analysis of the parsing format of date
[Hacker News Weekly] data visualization artifact; Top 10 Web hacker technologies; Postman supports grpc
[sword finger offer] serialized binary tree
LeetCode:39. 组合总和
What is the role of automated testing frameworks? Shanghai professional third-party software testing company Amway
opencv+dlib实现给蒙娜丽莎“配”眼镜
PC easy to use essential software (used)
[embedded] print log using JLINK RTT
自动化测试框架有什么作用?上海专业第三方软件测试公司安利
LeetCode:41. Missing first positive number
Philosophical enlightenment from single point to distributed
Tdengine biweekly selection of community issues | phase III