Ernie gram, an explicit and complete n-gram mask language model, implements explicit n-gram semantic unit knowledge modeling.

Model framework

from ERNIE 1.0 rise , Baidu researchers introduced Knowledge increases Study , adopt

Mask consecutive words 、

phrase、

named entity etc. Semantic knowledge unit ,

Achieve better pre training learning . This open source general semantic understanding model ERNIE-Gram Further more , Proposed Explicit 、 complete Of n-gram Mask language model , Implements explicit n-gram Semantic unit knowledge modeling .

ERNIE Multi granularity pre training semantic understanding technology

As the basic semantic unit of naturallanguageprocessing , Fuller language granularity learning can help the model to achieve stronger semantic understanding ：

• ERNIE-Gram Propose explicit complete n-gram Multi granularity mask language model , Synchronous modeling n-gram Inside and n-gram Between The semantic relationship of , Learning at the same time ** fine-grained （fine-grained） and coarse-grained （coarse-grained）** Semantic information
• ERNIE-Gram use Dual flow structure , In the process of pre training, the hierarchical prediction of single location and multi meaning is realized , Further enhance semantic knowledge learning

ERNIE-Gram Multi granularity pre training semantic understanding technology , stay Preliminary training (pre-training) Stage implements explicit multi granularity semantic signal learning , stay fine-tuning (fine-tuning) Phase adoption bert-style Fine tuning mode , Without increasing parameters and computational complexity , obtain 10 term English authoritative task SOTA. On the Chinese task ,ERNIE-Gram Include NLI、 Reading comprehension needs to be enriched 、 Multi level semantic understanding tasks are made public SOTA.

ERNIE-Gram Work has been NAACL-HLT 2021 As a long article , See... For more details link.

 ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding