How to solve the problem that iterative semi supervised training is difficult to implement in ASR training? RTC dev Meetup

Preface

「 Voice Processing 」 It is a very important scene in the field of real-time interaction , Launched on the sound network 「

RTC Dev Meetup Technical practice and application of voice processing in the field of real-time interaction

」 In the activity , From Microsoft Research Asia 、 Acoustic network 、 Technical experts of Sumi technology , Relevant sharing was conducted around this topic .

This paper is based on Sumi technology  NLP  Li Tian, the technical director, shared the contents in the activity . Official account 「

Sound network developer

」, Reply key words 「

DM0428

」 You can download activity related  PPT  Information .

01  Semi supervised training in  ASR  The necessity of the field

Universal  ASR  Although the accuracy of Chinese characters is already very high , But in specific scenarios （ Game scenario 、 Private chat scene 、 Group chat scene 、 Anchor scenario ） when , There is still a scenario mismatch problem , Because the universal  ASR  The application in these fields is relatively difficult , There are mainly the following problems .

1、 Mark the scarcity of resources

It is difficult to obtain the annotation of the corresponding scene , Usually, it is impossible to quickly obtain a large number of annotation samples required by business scenarios . Even if the sample acquisition is simple , However, it is still very difficult to obtain annotation samples , Because the marking cost is very high . When creating a project or determining the product direction , You will find that there are  ASR  The data problem should be solved before the task . In the past, when using phoneme and text splitting , The data volume is required to be small , Now, end-to-end technology is often used , Hold a candle to  1000  The amount of data started in an hour , Whether it is self labeling or with the help of well-known data companies , Before the product starts , The cost is hard to accept .

2、 Instability of dimension quality

Wake up 、Siri  Interaction and other scenarios , The user knows that the back end will transcribe , But in most business scenarios, people are concerned about  ASR  Transcription is imperceptible .

For example, with  Siri  When communicating , If  Siri  I didn't hear the speaker clearly , Then people will try again , Make the expression more clear . But the real business level , Most of the time, the customer doesn't know that the back end is  ASR  Transcription , For example, live broadcasting platform . It may provide audit level requirements , It is impossible to inform the anchor that the voice is being transcribed , You need to pronounce more clearly . The annotation quality caused by unclear enunciation and broken syntactic components is very unstable .

So how to solve these problems when labeling ？ For the US business , Because it covers a large number of similar social scenes throughout the Internet , Faced with a wide variety of data and specific terms , Therefore, it is very difficult to obtain such annotations , At the same time, it is difficult to guarantee the marking quality , But the data of the scene can be easily obtained from the homologous data , We believe that the semi supervised scheme is an ideal choice .

If you have ever touched  NLP  perhaps  CV, I believe you will have a clear definition of semi supervision . stay  ASR  This field , Especially based on end-to-end , At present, it is generally divided into two types ：Self-training  and  Pre-training, Others are less common , Or it can't be in at present  ASR  The field has achieved a good landing .

Self-training  The system mainly revolves around the well-known  Pseudo labeling. The core scheme is mainly based on  consistency regularization  Logic . In theory ,Pseudo label  It's actually  true label  A kind of noise , During model training , take  Pseudo label  and  true label  Train together , This itself is the process of training anti noise , It can make the model learn step by step .Pre-training  It's simple . If you do  NLP  You will know better from birth , It was originally to train the more appropriate representation of the corresponding field in the corresponding field . This task generally revolves around the reconstruction of the meaning or content of representation , No need for extra labels , This data can be constructed without labels / Having no artificially transcribed words  Pre-training  The training task of , Then use the artificially transcribed data of the corresponding scene to  ASR  Task training .

01  Semi supervised training in  ASR  The development of the field

1、Self-training

Generally speaking ,Self-training  From  CV. from  2013  Year of  Pseudo label ICML  Put forward for the first time  Pseudo label  since , Various new systems have emerged , Such as  2014  year  Learning with pseudo-ensembles（ The first system ）, take  Pseudo label  And model  Ensemble  To merge ;2016  year  Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning  Think  Pseudo label  Its own generation logic should also be different disturbances of the same model ;2017  year  Mean teachers are better role models: Weight-averaged consistency targets  Focus on how to generate higher quality labels , It uses model averaging to get better  teacher  Model , So as to ensure the quality of false labels .

As early as  2014  year 、2016  Two papers in , It has already been mentioned in  CV  Compare and learn from the popular fields in the , The formula argument in this paper is almost the same from many aspects , It can be said that the development of technology is a historical cycle .

2、Pre-training

Pre-training  Mainly focused on  NLP  field , Of course.  CV  There are also areas such as  ladder network  system , contain  Pre-training  Concept . however  Pre-training  The better developed fields are still  NLP. The core problem is  NLP  The underlying feature of is the character , This itself is a very discrete system , It is difficult to communicate with  CV  This dense data input for comparison .

In terms of this system ,NLP  After years of development , from  1994  Year of  N-gram-based  features , Based on the  NN  system , And then later on  NN  Generated by the design of the internal framework of the system  RNN  and  LSTM  And other language models ,2017  year  ELMO  Born in the sky , Until then  2018  year  transformer  The architecture appears . Now? , Whether it's  BERT  Or is it  GPT  Waiting  NLP  Various downstream businesses in the field have been fully verified .

3、ASR  Semi supervised development in the field

Generally speaking, according to  ASR  Its own times split it into two sections ：

① Based on phonemes / The era of text splitting ： In many cases, people still use  kaidi  As a business level  ASR  Underlying technical solutions . The semi supervised training logic of the scheme is , An acoustic model can be trained to  general  Phoneme model , Then through the downstream language model or  rescore  The model outputs the text required by the specific business , So as to achieve the function of partial semi supervision . From the process , It is more like a kind of transfer learning . But as the  Alex Graves  stay  2013  Years to complete  CTC  After my doctoral thesis , The end-to-end system began to emerge gradually . Two years later , EESEN  The team renewed  CTC  To the phoneme level , Make phonemes / The text splitting system returns briefly .

② End to end era ：LAS（listen attendance style）  The rise of the system , as well as  CTC/LAS + LM hybrid  The rise of the system , Make the end-to-end effect 、 data 、 Model quality and reasoning speed , Begin to surpass  Kaldi  Or traditional phonemes / Text split model architecture , The industry has also begun to step into an end-to-end era . The time sequence is  CTC,Deep speech,Listen,attend and spell,  as well as  Hybrid CTC/attention.

stay  2017  Years later , With  Watanabi  Put forward  CTC/attention hybrid  and  ESPNET  Release of the frame , The end-to-end system has been preliminarily improved and can be applied to various businesses in industry . It provides a set of the same Lattice  The same flexible combination decode frame ： be based on hypotheses route The design of the , Give follow-up shallow fusion  More flexible integration solutions . In fact, if you've ever used  ESPnet, You can see the whole  hypotheses  Path design is very flexible , Various technical solutions can be introduced to route Joint scoring or  rescore.

Because no longer use phoneme and other basic , And  CTC  and  Seq2Seq  The training cost is very high , In addition, it is difficult to obtain the actual annotation data , The short board that the end-to-end system relies on data has gradually become the core bottleneck of its implementation . If in the early days, especially  2015  year -2016  I worked in a big factory in  ASR, Your actual landing experience is , stay  1000  Consider end-to-end after hours .

thus , How to constrain end-to-end data requirements becomes a late problem （ from  2019  year -2020  Year begins ） Optimize end-to-end , And then solve the problem of end-to-end landing , It is also the core consideration of academia and industry . Since then , be based on  ASR  Of  Pre-training  and  Self-training  Began to step onto the stage of history . before , Although relevant research has been done , But the scope of influence is small , until  2019  Years and  2020  year ,Facebook AI  It is proposed that these two fields can be industrialized , Two papers with great development prospects were published , People began to pay attention to .

wav2vec: Unsupervised pre-training for speech recognition  yes  Facebook  Based on  Pre-training  Technical solution . The principle is the same as  word2vec  Very close to , Using the negative sampling technique to train a future time representation prediction task . Because the training results can be used as the characteristics of any audio downstream task , So this system is a very important audio technology foundation used by many large factories in the industry .

Self-training for end to end speech recognition  yes  Facebook AI  Of  Jacob  Team research , Aimed at a comprehensive analysis  Pseudo label  System for  ASR  The actual application effect . They gave  Pseudo label  The system is in English  ASR  On several core data sets in the field  strong baseline, And the first systematic exposition of  Pseudo label  The system is  ASR  Several core problems that need to be solved in the field landing .

4、Pre-training VS Self-training in ASR

stay  2020  year , As the number of customers gradually increases , More and more scenes are covered , We also face ： You need to do a separate... For some specific scenarios  ASR  structure , To obtain better model effect than competitive products . Simply use phonemes / Text structure , We can not achieve the desired effect by replacing the language model to meet the needs of various fields . But at the same time , Build your own end-to-end for each scenario individually  ASR, It is difficult to accept from the data annotation . So we began to consider the choice  Pre-training  still  Self-training.

Originally, we considered to choose similar systems of other large factories , such as  Pre-training  Of  wav2vec, But we tried many times  wav2vec  Actual operation of , The cost is very high , The downstream  Post-pretraining  Training in the corresponding field plus  Pre-training  The training time itself is also very long , As a result, the iteration cycle of the model will be prolonged . It is important to , stay  Pre-training+Post-pretraining  There is nothing at this stage  ASR  The output of the model , For new business scenarios that require rapid iteration , It's hard to accept .

Based on the above contradiction , We ultimately prefer to use... In our business  Self-training  Technical solution . because  Self-training  The technical scheme of can be evaluated for each training model , Use first and optimize later , This is a friendly system for business .

5、 In the near future  ASR  field  Self-training  Development track

Anchored  Self-training  After the goal , from  2020  Since, we have been conducting research and follow-up in this field . We found that , In this field, it is mainly  Facebook,Google, Mitsubishi   It has been done quite well , Others such as old brands  ASR  company  Nuance  And some colleges and universities will also publish some improvement plans or problem research for some specific problems . stay  2020  year , Their main research interests are as follows ：

(1) 2020  year

Facebook：

SELF-TRAINING FOR END-TO-END SPEECH RECOGNITION,

END-TO-END ASR: FROM SUPERVISED TO SEMI-SUPERVISED LEARNING WITH MODERN ARCHITECTURES,

ITERATIVE PSEUDO-LABELING FOR SPEECH RECOGNITION

The research context is   simple  Pseudo label  stay  CTC  In frame  strong baseline  And research ; simple  Pseudo label  stay  CTC/Attention hybrid  Architectural effects ; Multi round iteration  Pseudo label  The study of systems .

Google：

because  Google  Of  Iterative pseudo-labeling  stay  CV  The field has a very strong technical background , So as soon as they came up, they gave their multi round iterative formula  Pseudo label+model ensemble  programme ：Noisy Student Training, And take the year  Librispeech100 + 860 SOTA. Of course ,Iterative  There are many holes in training , In particular, the explosion in the number of data experiments caused by multiple rounds of iteration . This is clearly stated in our plan .

Mitsubishi ：

Iterative  Pattern , In the process, the first step is to  teacher  Carry out multiple rounds of  pseudo-labeling  Training , Every training one  pseudo-labeling, The interior needs to be labeled , Such multiple rounds will make the training very cumbersome . So from  2021  Year begins , We have also gradually seen in various fields  on-the-fly  The way . For example, Mitsubishi is  2012  Put forward in  MPL（ be based on  mean teacher  Evolved ）. however  on-the-fly  This means that real-time generation is required  label, and  ASR  Of  label  The generation quality is the same as  decode  The calculation of costs is directly related to . ordinary  CTC  Of  greedy search  Faster , However, the quality of the transcribed text produced by it is poor ; And the more common  shallow fusion  programme , Only multiple models are used for scoring  decode  Transcribe to produce words , It is basically impossible to generate in real time during training . So generally speaking ,on-the-fly  The final effect of the mode is actually not as good as  Iterative  Pattern .

other ：

Saleforce  Once “ The Renaissance ”,  The pseudo tag training is used again in  Essen  On the frame . Its label generation uses  CTC greedy search.Nuance  As the old  ASR  Technology manufacturers , By expounding  FixMatch  The theory interprets the theoretical essence of semi supervision, which is actually  Consistency Training.

(2) 2021  year

Mitsubishi ：

because  on-the-fly  The defects of the model , Mitsubishi in  2021  In published  advanced MPL, It's back  Iterative  Pattern . They will  teacher  Model and subsequent  on the flying  Split the training process , At the same time, it switches to a more robust audio effect  Conformer  frame . Finally, it surpassed  Google  Of  NST  programme , Become the second place at present .

Facebook：

Facebook AI  stay  2021  Used in  cache  Mechanism , Synchronize another process during model training  decode, If  cache decode  Full of , Just cut the training into  cache  Data and  label  Data for joint training ,N  After a step  catch  Empty , And then do it again  decode. so , although  Facebook AI  Said he is  on-the-fly  Pattern , But in essence, it is the concept of rounds . Its use  36  layer  transformer, Got it so far  Librispeech100+860  Of  SOTA, It can even be balanced  ESPnet  Direct training  Librispeech960  了 .

03  Our semi supervised solution solves the problem

1、Iterative or on-the-fly

It is in the effect demand and the conclusion of the current academic and industrial circles , Our technical direction is finally anchored  Iterative  Pattern .

2、Iterative  The problem of

but  Iterative  Pattern training is very cumbersome , Since the generation of pseudo tag data needs to be regenerated after each round of training , And to achieve good results , according to  Google  and  Facebook  Experience , Multiple iterations are required .

So each iteration has three problems , First of all , How to generate high-quality data on pseudo tags ？ This is essentially the simplest problem , We have all kinds of  decode  Algorithm , Use whichever algorithm is good . second , How to filter out high-quality pseudo tag data ？ Because we don't know which label is right , No matter how high the quality , There will be some problems , At this point, we need to study how to reduce the proportion of problems , What can be done to reduce . Third , Whole  Iterative  The biggest problem in the pattern is , How to balance labeled data and non labeled data .

Google  Of  NST  Our system needs five iterations , It means that the ratio of marked and unmarked in each round is different . The second round is probably  2:7, The third round is  1:3, stay  librispeech 100+860, This one has a label ： No label   Maintenance in  1：3  The upper and lower ratios are verified to be reasonable . But on different mission lines , The ratio is also different .Facebook  stay  Librispeech+LibriVox  The experimental results on the data set show that the ratio needs to be within  1:10  above . This leads to the final landing in the business , The cost of the experiment is huge . For example, there are five experiments ,  Each round of training requires multiple data experiments with different ratios , After the training, select the model for  decode  assessment , Then in the next round, multiple data experiments with different ratios will be carried out again , This iterates over five rounds . because  ASR  Training is expensive , The pace of each round of training is very painful .

in addition , At the limited dimension level , How to start the model cold ？ Generally speaking , The initial training data is labeled , Training data are very few . such as  Iterative  The initial tag data in is generally very small , Only of the available data  1/10  about , So how to carry out cold start has become a core problem .

04 Improved NLPL  Solution

Based on these questions , We have come up with our own solutions , Published in  Improved noisy Iterative Pseudo-Labeling for Semi-superivised Speech Recogntion  in . Now let's briefly explain our solution in advance .

1、 Model framework

from  2020  Years later , We will no longer use  Kaldi  System , Instead, you switch to a class  ESPnet  Self research framework . On the model frame , about  CTC  The front end of the  sharedEncoder  and  LAS  Of  decoder, We all use  transformer, chart  1  On the left is  Watanabi  stay  CTC/Attention hybrid  The figures in that paper , On the right is an introduction to the model framework , Model parameters ,SharedEncoder  There was one before  subLayer, It's using  2  layer  (3

3+512)  Of  CNN, Step by step to  2, This may be related to  ESPnet  The frames in are slightly different , But it's basically the same .ransformer  We currently use  12

8  Of  transformer,512  dimension ,FFN  yes  2048, This is similar to most  formerbase  The model is almost the same . in addition ,AttentionDecoder  We're going to use  6  layer  transformer, Its parameter configuration is similar to  Encoder  It's the same . Language model ,LT  people ！ Inserted  4  We added an extra  6  Layer of  transformer  Language model , Other parameter configurations are the same as  BERT  It's the same ,12  head ,768dims,FFN  by  3072, This is the overall model framework .

from  2020  Years later , We will no longer use  Kaldi  System , Instead, you switch to a class  ESPnet  Self research framework . On the model frame , about  CTC  The front end of the  sharedEncoder  and  LAS  Of  decoder, We all use  transformer, chart  1  On the left is  Watanabi  That article  CTC/Attention hybrid  Graph in the paper , On the right is an introduction to our model framework . Model parameters ,SharedEncoder  Of  sublayer  At present, it is  2  layer  (3*3+512)  Of  CNN, Step by step to  2,Transformer  We currently use  12  layer  8  head ,512  dimension ,FFN  yes  2048, This is similar to most  Transformer-based  The acoustic model is almost the same . in addition ,AttentionDecoder  We're going to use  6  layer  transformer, Its parameter configuration is similar to  Encoder  It's the same .

For the language model , We added an extra  6  Layer of  transformer  Language model , Other parameter configurations are the same as  BERT  It's the same ,12  head ,768dims,FFN  by  3072.

■ chart  1

2、 Other general settings

Our experimental data are  Librrispeech 100+860,100  As marked data ,860  As dimensionless data .LM  The data is  Librispeech  Your training data , And the official  800W  Text corpus . Our vocal features are  100  dimension  Fbank+3  dimension  pitch. To reduce the number of text labels , We used  BPE, hold  word  The quantity is reduced to  7002  individual  pieces  To reduce the final output , At the same time accelerate  CTC  Training for .

Training configuration involves learning rate , Learning rate and  transformer  be similar , But there are differences , Is in the  decay  To the last position , We'll advance  5000step decay  To the final stable value , Then hold it slowly for a while . This is directly related to the following technologies to maintain model stability , Let it train steadily for a period of time during that period , So that the average model can keep up with .

3、 How to generate false labels on unlabeled data

At present, it is common in the industry  decode  Algorithmic and relatively high-quality methods are  shadow fusion  and  deep fusion  system . We used  shadow fusion, And the acoustic model  CTC、LAS  as well as  LM  Merge to search ,bean size  by  50. The general process is the same as  ESPNET  almost , But we have two small changes ：

The first is that we use  CTC  The way of greedy search is to judge the end of the sentence , and  ESPNET  It's not done , It has its own  end detact  Algorithm .

The second is that we will not prune the path too much , Instead, keep as many paths as possible .

4、 How to select high-quality pseudo tag data for the next round of semi supervised training

When generating pseudo tags , In fact, the quality of many data is not flattering , Especially the early training , such as  NST  perhaps  Iterative Labeling  The first or second round of , At this time, the model is  librispeech dev  and  test  Upper  WER  May be close to  9  perhaps  10  More than one point .

In this case ,Google  and  Facebook  Adopt the method of rough sorting to take the percentile , Be similar to  ESPNET  Medium  hypothesis  The score of , And then in  decode  During the process, the probability is added , Rank probabilities from small to large , Then take one of them  90%. There may be a cliff like confidence rate , Like the front  85%  The probability distribution of the data is very similar , And then in  85%～95%  The location of , The probability suddenly shows a very big difference , The probability of falling to more than a few possible points . In response to the above problems , We use the method of distribution test to extract samples ： Let us first assume that it obeys the Gaussian distribution , Then only the bilateral confidence intervals of Gaussian distribution are preserved  90%  perhaps  95%  For training . Here is the bilateral confidence interval  90%/95%, Does not mean data retention  90%  and  95%, But in the case of Gaussian distribution, keep the data in the confidence interval , So it's probably less than direct retention  90% Data .

5、 mark / How to balance the ratio of unmarked data , Only in this way can the model not be over fitted to label data without label data

mark / How to balance the ratio of unlabeled data is the biggest problem in multi round iterative semi supervised training , All the previous studies did not show how to conduct proportional screening , Only the approximate proportion of corresponding tasks is given ,Facebook  What they do is  Librispeed 960+LibriVOX, Its ratio is  1:10~1:54  Between .Google  yes  Librispeech 100 +800, Scale in  1:3  about .

The above opinions can not guide the actual production to determine the proportion of land use . For example, live broadcast scenes  ASR, With  100  Hours as the starting price , At the same time, it may be easy to obtain many homologous unlabeled data . But in what proportion should these unlabeled data and labeled data be put together , So that the model will not be trained to unlabeled data ; How to train the model to ensure its stability and better effect , This will require endless data experiments . Of course , If there are enough machine resources in the company , It is indeed possible to do these experiments , But most of the time, people are not like  Google  and  Facebook  There are so many machines , It can be directly and violently exhausted .

So how can we get guidance from each business line at this time ？ We are  Librispeech 100/860  Detailed experiments and qualitative and quantitative analysis were carried out , Got a guide , At present, this guidance is very accurate , It can teach you how to choose data balance . Let's make a hypothesis here , This is directly related to why we should do semi supervised training of false labels . We think that when training pseudo tags , Because tagged data and unlabeled data are mixed together , So for some pseudo tag data , We don't know if it's marked correctly , Model training should be made as much as possible on certain characteristics “ conservative ”, Don't over fit to the wrong data or trailer data . But it also ensures a certain diversity of samples , Because if you are completely conservative , Model training will fall into what it thinks is the optimal result of the data level , And then step into the local optimal solution . Multiple rounds of iterative training will exacerbate this process , This leads to model over training and over fitting .

To identify where to be conservative , Where to ensure diversity , We divide the data into three portrait dimensions , The first portrait dimension is audio length , The second portrait dimension is text /pieces  length , The third dimension is the distribution of the tag itself . The problem can be transformed into , In which dimensions should we try to keep the training conservative , Which dimensions should ensure the diversity of samples as much as possible . Based on this , We conducted a large-scale experiment , Every time a new pseudo tag is generated , We will according to different proportions , Build multiple training samples  candidate, That is, the alternative set , This  candidate  Each batch of training data in . Before each round of training , We all share every catenary  cadidate  Same as our last training data   Compare these three dimensions , And for all  candidate  Rank . such as  1:2  Of  candidate  Rank in three dimensions with the upstream ,1：4  Of  candidate  There will also be a ranking ,1:5  and  1:6  There will also be a ranking , wait .

On the evaluation ranking scheme , because  frame lenth  and  pieces length  Is a single dimensional statistic , So we used  KS  test . but  label  The distribution itself is multidimensional , So let's normalize  TF, Then the Euclidean distance is used to evaluate the distribution difference between the current round data and the previous round data , And for each  candidate  ranking .

After a lot of experiments , Found a very clear rule , Namely  pieces  The smaller the difference of distribution itself , Bigger  frame lenth  Distribution differences and  pieces length  The difference in distribution will generally lead to a better new round of model effect . The above logic can be described as a general paradigm , Pictured  2  Shown .

■ chart  2

6、 How to ensure that the model will not be over fitted to the wrong pseudo label in model training  trick

This is a key point we found in the whole system . Here we have two dimensions . The first dimension is the data dimension , We joined in  specAug  and  specAug++ Make the whole data more generalized . At the same time, at the model level , Be similar to  MPL, We will generate  online  and  offline  Generation , Choose... In the early stage  online  Result , Post selection  offline  Result , Generally speaking, after the fifth round  offline  The result will be stable higher than  online  Result . in addition , We will also carry out  dropout  promote , about  dropout  From  0.1  Gradually upgrade to  0.3, because   Pseudo tag training   There is a great risk of over fitting , But it's basically up to  0.4  There will be no new income in the future .

7、 Under a limited number of labeled samples , How to carry out the cold start supervision training of the model can obtain the best effect

We also used two-stage training . The first stage of training starts with  dropout0.1 30epoch  Match to the second level  dropout0.13 100epoch  The best effect . The specific experimental results are shown in the figure  3  Shown . It also shows a problem , In cold start, you should start with a few  epoch, The smaller  dropout, Fast fit target , Then raise  dropout, Let it be a relatively generalized training configuration , Train more rounds , Make the model optimal . This cold start mode can basically be compared with  Google  Of  NST  The cold start result of the system model is flat .

■ chart  3

Finally, the whole  improved NIPL  The final effect of . At present, the deadline for submission is  interspeech 2022  Look at , stay  Librispeech 100+860  At present, there are two companies that are better than us , The first is Mitsubishi  MPL  Of  conformer  yes  3.8%/8.2%. But if the control variable is the same transformer, Mitsubishi has only  4.8%/10.1%, And we are  3.93%/9.59%. The other is  Facebook  Of  simIPL, its  36  layer  transformer  It can be done  3.8%/7.5%, And no language model is required , If you add a language model and  rescore  It can be done  2.7%/5.2%. This effect already belongs to the effect beyond our cognition . Because we trained  960  The data of ,ESPnet librispeech 960  The supervision training results in  96.96  Should be  3.04%, It means  Facebook  no need  860  The data of , only  100  Of  label  You can do that  2.7%/5.2%.

Finally, the whole  improved NIPL  The final effect of . At present, the deadline for submission is  interspeech 2022  Look at , stay  Librispeech 100+860  At present, there are two companies that are better than us , The first is Mitsubishi  MPL  Of  conformer  yes  3.8%/8.2%. But if the control variable is the same  transformer, Mitsubishi has only  4.8%/10.1%, And we are  3.93%/9.59%. The other is  Facebook  Of  simIPL, its  36  layer  transformer  It can be done  3.8%/7.5%, And no language model is required , If you add a language model and  rescore  It can be done  2.7%/5.2%. This effect already belongs to the effect beyond our cognition . Because we trained  960  The data of ,ESPnet librispeech 960  The supervision training results in  96.96  Should be  3.04%, It means  Facebook  no need  860  The data of , only  100  Of  label  You can do that  2.7%/5.2%.

05  Q & A

1、 contrast  WER  What's the effect ？

our  test clean  yes  3.93,test other  yes  9.59, But then we went on NIPL Training rounds 7 and 8 ,test other  It can be reduced . although test clean  Still maintained at  3.93, but  test other  So far, it has been reduced to about  9.3. Mitsubishi's  conformer  yes  3.8%/ 8.2%, Better than ours 3.93  low , But their  transformer  yes  4.8%/10.1%.Facebook  Of  simIPL  yes  3.8%/7.5%, about  Facebook simIPL  We expressed a little disbelief , The effect is a little scary . So we should be the third in the world , Than  Google  stay  2020  The article published in  NST  Better .

2、 Introduce to you  CTC  Use

CTC  When it first appeared , Because of the difficulty of training optimization , The requirements for data volume are also strict , So at that time  CTC  The use of is all some strange skills . As mentioned above  ESSEN, hold  CTC  For training phonemes , Then I still pick up the same as everyone else  WFST. Because the number of phonemes is relative to  word  Much smaller , Greatly reduced  CTC  The difficulty of training , So that it can be the same in some fields  MMI,LFMMI  And so on . Directly naked  CTC  End to end  ASR  Data costs can be very high .

If you are in the  2020  Ask this question in , I will recommend you to try in the new business  ESSEN  project . But now it's  2022  Years. ,CTC  Great changes have taken place in the industrial use of .Watanabi  The paper tells you ,CTC  and  LAS hybrid  This system can have a very good effect , And the data quality will not be the same as before  CTC  That requires so much , because  LAS  The system has many optimization techniques that can be used to help train . therefore  CTC LAS  It is a relatively standard use scheme at present . If you don't have your own  ASR  Training platform , I suggest you try  ESPnet/Wenet, If flow identification is the core business demand ,Wenet  As the first choice .

Activity Notice

「RTC Dev Meetup -  Hangzhou station 」, We will focus on big front-end technology , The invitation comes from

Acoustic network 、 Ant group and Hikvision

Technical experts , Share with us the business architecture and cross end practices in the field of real-time interaction in the era of big front-end .

Action is better than action , Scan the QR code or click

here

Sign up ！