Text error correction -- crisp model

Text correction –CRASpell Model

CRASpell: A Contextual Typo Robust Approach to Improve Chinese Spelling Correction This paper was published in 22 year ACL, stay Chinese spelling correction (CSC) The task is SOTA. be based on bert In the pre training model CSC Our model has two limits ：
(1) The model does not work well on multi error text , Usually in misspelled text , Misspelled characters occur at least 1 Time , This will bring noise , This kind of noisy text leads to the performance degradation of multi wrong word text .
(2) because bert Mask task , These models over correct the useful expressions of high-frequency words .
CRASpell Model each training sample to construct a noisy sample ,correct The model is based on the output which is more similar to the original training data and the noise sample output , To solve the over correction problem , A copy mechanism is combined to enable our model to select input characters when error correction and input characters are valid according to the given context .
The address of the article is ： article
The code address is ：code

Model

Task description

The purpose of Chinese spelling correction is to detect and correct spelling errors in text . Usually expressed as $\mathbf{X}=\{x_1,x_2,\dots ,x_n\}$ It's a length of $n$ Contains misspelled text ,$\mathbf{Y}=\{y_1,y_2,\dots ,y_n\}$ It's a length of $n$ Correct text for , Model input $\mathbf{X}$ Generate correct text $\mathbf{Y}$.

CRASpell Model

On the left is Correction Model , On the right is Noise Model , The following describes the model in detail .
(1) Correction Module
Given the input text $\mathbf{X}=\{x_1,x_2,\dots ,x_n\}$ obtain embedding vector $\mathbf{E}=\{e_1,e_2,\dots ,e_n\}$, Each of these characters $x_i$ Corresponding embedding The vector is written as $e_i$, take $\mathbf{E}$ Input to Transformer Encoder Get in hidden state matrix $\mathbf{H}=\{h_1,h_2,\dots ,h_n\}$, among $h_i\in {\mathbf{R}}^{768}$ Is the character $x_i$ after Transformer Encoder Get the feature .
(2) Generative Distribution
$\mathbf{X}=\{x_1,x_2,\dots ,x_n\}$ after Transformer Encoder Get the eigenvector $\mathbf{H}=\{h_1,h_2,\dots ,h_n\}$, Through a forward linear layer and a softmax Layer gets each character token Generation probability of , The formula is as follows :

$$p_g=softmax(W_g{h}_i+b_g)$$
among $W_g\in {\mathbf{R}}^{n_v\times 768}$,$b_g\in {\mathbf{R}}^{768}$,$n_v$ Is the size of the pre training model vocabulary .

(3) Copy Distribution
$x_i$ Of copy distribution $p_c\in \{0,1{\}}^{n_v}$ yes $x_i$ In the dictionary $idx(x_i)$ Of one-hot Express , The specific expression is as follows ：

(4) Copy Probability
Copy Probability Is in the model diagram Copy Block The output in $\omega \in\Large\boldsymbol{R}$, namely transformers encoder The obtained hidden layer eigenvector $h_{i}$ Through two forward linear layers and one layer normalization obtain $\omega$, The formula is as follows ：

$$\begin{gathered} h_c=W_{ch}{f}_{ln}(h_i)+b_{ch} \\ {h}_c^{{}^{\prime }}=f_{ln}(f_{act}(h_c)) \\ \omega =Sigmoid(W_c{h}_c^{{}^{\prime }}) \end{gathered}$$
among $W_{ch}\in {\mathbf{R}}^{768\times d_c}$, $b_{ch}\in {\mathbf{R}}^{d_c}$,$W_c\in {\mathbf{R}}^{d_c\times 1}$,$f_{ln}$ yes layer normalization,$f_{act}$ Is the activation function , The activation function used in the code is glue. See the code for details

Copy Block Output probability $p$ Combined generation Generative Distribution $p_g$ and Copy Distribution $p_c$
$$p=\omega \times p_c+(1-\omega )\times p_g$$
Prior to CSC The difference between models is that ,CRASpell The model takes into account... In the final generation output of the model Copy Probability $p_c$, Make the model valid in input characters but not the most suitable BERT There are more opportunities to choose input characters , Avoid overcorrection .
(5) Noise Modeling Module
Noise Modeling Module The correction model produces a similar distribution for the original context and the noise context to solve the problem of context misspelling interference . As shown on the right side of the model diagram above ,Noise Modeling Module It can be roughly divided into the following processes ：
a. According to the input sample $\mathbf{X}$ Generate noise context $\tilde{\mathbf{X}}$
b. Noise context $\tilde{\mathbf{X}}$ As input, get Transformer Encoder Get the hidden eigenvector $\tilde{\mathbf{H}}$
c. Based on hidden eigenvectors $\tilde{\mathbf{H}}$ Generate generate distribution $\widetilde{p_g}$
d. The generated distribution is similar to that generated by the calibration model . The generated distribution is similar to the distribution generated by the calibration model minimizing the bidirectional Kullback-
Leibler divergence reflect , The formula is as follows ：
$${\mathcal{L}}_{KL}=\frac{1}{2}({\mathcal{D}}_{KL}(p_g\Vert \widetilde{p_g})+{\mathcal{D}}_{KL}(\widetilde{p_g}\Vert p_g))$$
remarks ：Noise Modeling Module Only during training , Model reasoning only uses correction networks
Noisy Block
The following describes adding noise data to data . The noise sample is generated by replacing the characters of the original training sample . In the process of replacing characters, only the misspelled character context is replaced $d_t$ Word , If the training sample is not misspelled , This sample is not replaced to generate noise samples . As shown in the figure below ：

$d_t$ Select the experimental results

We replace each selected position with a similar character based on the publicly available confusion set . say concretely , We choose the word at the position to replace
(i) 70% The replacement randomly selects phonetically similar characters
(ii) 15% Random selection of glyph like characters
(iii) 15% A random selection of alternatives from the vocabulary .

Loss

Given a training sample $(\mathbf{X},\mathbf{Y})$,$\mathbf{X}$ Is an input error sample ,$\mathbf{Y}$ Is to correct the sample , Correct sample for each calibration ${\mathbf{Y}}_{\mathbf{i}}$ Of loss by
$${\mathcal{L}}_c^i=-\log (p({\mathbf{Y}}_{\mathbf{i}}|\mathbf{X}))$$
among $p$ by
$$p=\omega \times p_c+(1-\omega )\times p_g$$
See the introduction above for details .
Model loss by $\mathcal{L}$
$${\mathcal{L}}^i=(1-{\alpha }_i){\mathcal{L}}_c^i+{\alpha }_i{\mathcal{L}}_{KL}^i$$
among ${\alpha }_i$

among $\alpha$ yes ${\mathcal{L}}_c$ and ${\mathcal{L}}_{KL}$ The trade-off factor . The constructed noise samples themselves will not participate in the training process , It will only participate as a context . This strategy aims to ensure that the constructed noise data will not change the proportion of positive and negative samples in the training corpus .

experimental result

Later supplement sighan The result on the dataset .
CRASpell Model experimental results