当前位置:网站首页>Explain Bleu in machine translation task in detail
Explain Bleu in machine translation task in detail
2022-07-07 07:09:00 【aelum】
Catalog
One 、 n n n Metagrammar (N-Gram)
n n n Metagrammar (n-gram) Refers to the text continuity The emergence of n n n individual Morpheme . When n n n Respectively 1 , 2 , 3 1,2,3 1,2,3 when ,n-gram It's also called unigram( Unary grammar )、bigram( Binary grammar ) and trigram( Ternary grammar ).
n n n The meta grammar model is based on n − 1 n-1 n−1 A probabilistic language model of order Markov chains ( That is, only the former is considered n − 1 n-1 n−1 When words appear , The probability of the latter word ):
unigram: P ( w 1 , w 2 , ⋯ , w T ) = ∏ i = 1 T P ( w i ) bigram: P ( w 1 , w 2 , ⋯ , w T ) = P ( x 1 ) ∏ i = 1 T − 1 P ( w i + 1 ∣ w i ) trigram: P ( w 1 , w 2 , ⋯ , w T ) = P ( x 1 ) P ( x 2 ∣ x 1 ) ∏ i = 1 T − 2 P ( w i + 2 ∣ w i , w i + 1 ) \begin{aligned} \text{unigram:}\quad&P(w_1,w_2,\cdots,w_T)=\prod_{i=1}^T P(w_i) \\ \text{bigram:}\quad&P(w_1,w_2,\cdots,w_T)=P(x_1)\prod_{i=1}^{T-1} P(w_{i+1}|w_i) \\ \text{trigram:}\quad&P(w_1,w_2,\cdots,w_T)=P(x_1)P(x_2|x_1)\prod_{i=1}^{T-2} P(w_{i+2}|w_{i},w_{i+1}) \\ \end{aligned} unigram:bigram:trigram:P(w1,w2,⋯,wT)=i=1∏TP(wi)P(w1,w2,⋯,wT)=P(x1)i=1∏T−1P(wi+1∣wi)P(w1,w2,⋯,wT)=P(x1)P(x2∣x1)i=1∏T−2P(wi+2∣wi,wi+1)
Two 、BLEU(Bilingual Evaluation Understudy)
2.1 BLEU Definition
BLEU( Pronunciation and words blue identical ) It was first used to evaluate the results of machine translation , But now it has been widely used to evaluate the quality of output sequences in many applications . For the prediction sequence pred
Any of the n n n Metagrammar , BLEU This is the assessment of n n n Whether the meta syntax appears in the tag sequence label
in .
BLEU The definition is as follows :
BLEU = exp ( min ( 0 , 1 − len(label) len(pred) ) ) ∏ n = 1 k p n 1 / 2 n \text{BLEU}=\exp\left(\min\left(0,1-\frac{\text{len(label)}}{\text{len(pred)}}\right)\right)\prod_{n=1}^kp_n^{1/2^n} BLEU=exp(min(0,1−len(pred)len(label)))n=1∏kpn1/2n
among len(*) \text{len(*)} len(*) Represents a sequence ∗ * ∗ The number of lexical elements in , k k k Used to match the longest n n n Metagrammar ( Constant access 4 4 4), p n p_n pn Express n n n The accuracy of meta grammar .
To be specific , Given label
: A , B , C , D , E , F A,B,C,D,E,F A,B,C,D,E,F and pred
: A , B , B , C , D A,B,B,C,D A,B,B,C,D, take k = 3 k=3 k=3.
First of all to see p 1 p_1 p1 How to calculate . We will first pred
Each of the unigram It's all figured out : ( A ) , ( B ) , ( B ) , ( C ) , ( D ) (A),(B),(B),(C),(D) (A),(B),(B),(C),(D), then label
Each of the unigram It's all figured out : ( A ) , ( B ) , ( C ) , ( D ) , ( E ) , ( F ) (A),(B),(C),(D),(E),(F) (A),(B),(C),(D),(E),(F), Then see how many matches there are between them ( Cannot match repeatedly , That is, one-to-one correspondence must be maintained ). It can be seen that there are 4 4 4 A match , and pred
There's a total of 5 5 5 individual unigram, therefore p 1 = 4 / 5 p_1=4/5 p1=4/5.
Look again. p 2 p_2 p2 How to calculate . We will first pred
Each of the bigram It's all figured out : ( A , B ) , ( B , B ) , ( B , C ) , ( C , D ) (A,B),(B,B),(B,C),(C,D) (A,B),(B,B),(B,C),(C,D), then label
Each of the bigram It's all figured out : ( A , B ) , ( B , C ) , ( C , D ) , ( D , E ) , ( E , F ) (A,B),(B,C),(C,D),(D,E),(E,F) (A,B),(B,C),(C,D),(D,E),(E,F), Then see how many matches there are between them . It can be seen that there are 3 3 3 A match , and pred
There's a total of 4 4 4 individual bigram, therefore p 2 = 3 / 4 p_2=3/4 p2=3/4.
Finally, let's see p 3 p_3 p3 How to calculate . We will first pred
Each of the trigram It's all figured out : ( A , B , B ) , ( B , B , C ) , ( B , C , D ) (A,B,B),(B,B,C),(B,C,D) (A,B,B),(B,B,C),(B,C,D), then label
Each of the trigram It's all figured out : ( A , B , C ) , ( B , C , D ) , ( C , D , E ) , ( D , E , F ) (A,B,C),(B,C,D),(C,D,E),(D,E,F) (A,B,C),(B,C,D),(C,D,E),(D,E,F), Then see how many matches there are between them . It can be seen that only 1 1 1 A match , and pred
There's a total of 3 3 3 individual trigram, therefore p 3 = 1 / 3 p_3=1/3 p3=1/3.
So in this case BLEU The score is
BLEU = exp ( min ( 0 , 1 − 6 / 5 ) ) ⋅ p 1 1 / 2 ⋅ p 2 1 / 4 ⋅ p 3 1 / 8 = e − 0.2 ⋅ ( 4 5 ) 1 / 2 ⋅ ( 3 4 ) 1 / 4 ⋅ ( 1 3 ) 1 / 8 ≈ 0.5940 \begin{aligned} \text{BLEU}&=\exp(\min(0,1-6/5))\cdot p_1^{1/2}\cdot p_2^{1/4}\cdot p_3^{1/8} \\ &=e^{-0.2}\cdot \left(\frac45\right)^{1/2}\cdot \left(\frac34\right)^{1/4}\cdot\left(\frac13\right)^{1/8} \\ &\approx0.5940 \end{aligned} BLEU=exp(min(0,1−6/5))⋅p11/2⋅p21/4⋅p31/8=e−0.2⋅(54)1/2⋅(43)1/4⋅(31)1/8≈0.5940
2.2 BLEU Discussion
according to BLEU The definition of , When the prediction sequence is exactly the same as the tag sequence ,BLEU The value of is 1 1 1. On the other hand , because e x > 0 e^x>0 ex>0 And p n ≥ 0 p_n\geq0 pn≥0, So there is
BLEU ∈ [ 0 , 1 ] \text{BLEU}\in[0,1] BLEU∈[0,1]
BLEU The closer the value of 1 1 1, It means the better the prediction effect ;BLEU The closer the value of 0 0 0, It means the worse the prediction effect .
Besides , because n n n The longer the metagrammar, the more difficult it is to match , therefore BLEU For longer n n n The accuracy of meta syntax assigns greater weight ( Fix a ∈ ( 0 , 1 ) a\in(0,1) a∈(0,1), be a 1 / 2 n a^{1/2^n} a1/2n Will follow n n n To increase by ). and , Because the shorter the prediction sequence is p n p_n pn The higher the value , So the coefficient exp ( ⋅ ) \exp(\cdot) exp(⋅) This term is used to punish shorter prediction sequences .
2.3 BLEU Simple implementation of
import math
from collections import Counter
def bleu(label, pred, k=4):
# Let's assume that the input label and pred Word segmentation has been carried out
score = math.exp(min(0, 1 - len(label) / len(pred)))
for n in range(1, k + 1):
# Use hash table to store label All of the n-gram
hashtable = Counter([' '.join(label[i:i + n]) for i in range(len(label) - n + 1)])
# The number of successful matches
num_matches = 0
for i in range(len(pred) - n + 1):
ngram = ' '.join(pred[i:i + n])
if ngram in hashtable and hashtable[ngram] > 0:
num_matches += 1
hashtable[ngram] -= 1
score *= math.pow(num_matches / (len(pred) - n + 1), math.pow(0.5, n))
return score
for example :
label = 'A B C D E F'
pred = 'A B B C D'
for i in range(4):
print(bleu(label.split(), pred.split(), k=i + 1))
# 0.7322950476607851
# 0.6814773296495302
# 0.5940339360503315
# 0.0
References
边栏推荐
- Under what circumstances should we consider sub database and sub table
- sql中对集合进行非空校验
- 剑指offer-高质量的代码
- 如何给目标机器人建模并仿真【数学/控制意义】
- libcurl返回curlcode说明
- Graduation design game mall
- How to do sports training in venues?
- Answer to the first stage of the assignment of "information security management and evaluation" of the higher vocational group of the 2018 Jiangsu Vocational College skills competition
- from . onnxruntime_ pybind11_ State Import * noqa ddddocr operation error
- 非父子组件的通信
猜你喜欢
From zero to one, I will teach you to build the "clip search by text" search service (2): 5 minutes to realize the prototype
Release notes of JMeter version 5.5
Brand · consultation standardization
Abnova membrane protein lipoprotein technology and category display
Basic introduction of JWT
Take you to brush (niuke.com) C language hundred questions (the first day)
$refs:组件中获取元素对象或者子组件实例:
MATLAB小技巧(29)多项式拟合 plotfit
毕业设计游戏商城
Use of completable future
随机推荐
Distributed ID solution
mysql查看bin log 并恢复数据
【mysqld】Can't create/write to file
Matlab tips (29) polynomial fitting plotfit
sqlserver多线程查询问题
impdp的transform参数的测试
Answer to the second stage of the assignment of "information security management and evaluation" of the higher vocational group of the 2018 Jiangsu Vocational College skills competition
AddressSanitizer 技术初体验
How to model and simulate the target robot [mathematical / control significance]
2018年江苏省职业院校技能大赛高职组“信息安全管理与评估”赛项任务书第二阶段答案
toRefs API 与 toRef Api
Multithreading and high concurrency (9) -- other synchronization components of AQS (semaphore, reentrantreadwritelock, exchanger)
【NOI模拟赛】区域划分(结论,构造)
7天零基础能考证HCIA吗?华为认证系统学习路线分享
异步组件和Suspense(真实开发中)
一文带你了解静态路由的特点、目的及配置基本功能示例
JWT的基础介绍
Jetpack compose is much more than a UI framework~
详解机器翻译任务中的BLEU
多线程与高并发(9)——AQS其他同步组件(Semaphore、ReentrantReadWriteLock、Exchanger)