当前位置:网站首页>Oppo Xiaobu launched Obert, a large pre training model, and promoted to the top of kgclue
Oppo Xiaobu launched Obert, a large pre training model, and promoted to the top of kgclue
2022-07-05 12:41:00 【Zhiyuan community】
In recent days, ,OPPO Xiaobu assistant team and machine learning department jointly completed a billion parameter model “OBERT” Pre training , Business achievements 4% The above promotion ; In the industry comparison evaluation ,OBERT Leap to the benchmark of Chinese language understanding CLUE1.1 Fifth in the overall list 、 Large scale knowledge map Q & A KgCLUE1.0 Top of the list , Enter the first tier on the billion level model , Scores of multiple subtasks and top ranking 3 The effect of the ten billion parameter model is very close , The parameter quantity is only one tenth of the latter , It is more conducive to large-scale industrial applications .
CLUE1.1 General list , common 9 Subtask
KgCLUE1.0, Knowledge map Q & a list
The technology is completely self-developed , Xiaobu promoted the landing of a billion level pre training large model
The emergence of large-scale pre training model , by natural language processing The task brings a new solution paradigm , It has also significantly improved all kinds of NLP The benchmark effect of the task . since 2020 year ,OPPO Xiaobu assistant team began to explore and apply the pre training model , from “ It can be industrialized on a large scale ” From the angle of , It has successively researched 100 million 、 Pre training model with parameters of 300 million and 1 billion OBERT.
Pre training model development & Application plan
Thanks to the low cost of data acquisition and the powerful migration ability of language models , at present NLP The main task of pre training is language model based on distributed hypothesis . Here it is , The Xiaobu assistant team chose the downstream natural language understanding class (NLU) There is better effect on the task MLM, And use course learning as the main pre training strategy , From easy to difficult, step by step , Improve training stability . First, the above is verified on the 100 million level model mask The effectiveness of the strategy , Its Zero-shot The effect is significantly better than open source base Level model , Downstream applications have also yielded benefits , Then it is applied to the billion level model training .
ZeroCLUE The list
It is worth mentioning that , From the experimental results of open source work , The greater the number and content diversity of corpus , The effect of downstream tasks will be improved . Based on the previous exploration and attempt , Billion OBERT The model was cleaned and collected 1.6 TB Level corpus , adopt 5 Kind of mask The mechanism learns language knowledge from it , The content includes encyclopedia 、 Community Q & A 、 News, etc , The scenario involves intention understanding 、 Multiple rounds of chat 、 Text matching, etc NLP Mission .
Strengthen application innovation , Xiaobu continues to plough deeply NLP technology
CLUE( Chinese language understanding assessment set ) The list is one of the most authoritative natural language understanding lists in the Chinese field , Set up, including classification 、 Text similarity 、 reading comprehension 、 Context reasoning, etc 10 Subtask , To promote NLP The continuous progress and breakthrough of training model technology .
NLP( natural language processing ) Technology is known as Artificial intelligence crown jewels . As the core of AI cognitive ability ,NLP yes AI One of the most challenging tracks in the field , Its purpose is to make computers have human hearing 、 say 、 read 、 The ability to write, etc , And use knowledge and common sense to reason and make decisions .
Xiaobu assistant was released on 2019 year , To 2021 end of the year , It has accumulated 2.5 Billion devices , The number of monthly live users has exceeded 1.3 Billion , The number of monthly interactions reached 20 Billion , Become the first mobile phone voice assistant with hundreds of millions of live users in China , It has become the representative of the new generation of intelligent assistants in China .
stay NLP Technical aspects , Little cloth assistant experienced from rule engine 、 Simple model to strong deep learning , And then to several stages of the pre training model . after 3 The development of , Assistant Xiao Bu is NLP The technology field has reached the industry leading level , this OBERT At the top of CLUE 1.1 Top five 、KgCLUE 1.0 Top of the list , It is the best proof of the precipitation and accumulation of Xiaobu assistant technology .
pass a competitive examination CLUE 1.1 And top the list KgCLUE 1.0 Ranking List , It mainly benefits from three aspects : One is to use the massive data accumulated by Xiaobu assistant , Get spoken language data , Promote the algorithm model to have a better understanding of the language of the intelligent assistant scene ; Second, maintain an open growth mindset , Follow up the latest developments in academia and industry and put them into practice ; Third, firmly invest in the direction of the latest pre training model , Do technical accumulation bit by bit , Explore landing applications again and again .
future , Xiaobu assistant team will combine the characteristics of intelligent assistant scene , Continuous optimization of pre training techniques , Deep tillage NLP, Use technology such as model lightweight to accelerate the landing of large models , And continue to explore AI Combination with active emotion , Make intelligence more humanized , In the era of the integration of all things , Help to promote AI To shine , help AI Moisten things silently into people's future digital intelligence life .
边栏推荐
- 前几年外包干了四年,秋招感觉人生就这样了..
- The evolution of mobile cross platform technology
- MySQL constraints
- 只是巧合?苹果 iOS16 的神秘技术竟然与中国企业 5 年前产品一致!
- Resnet+attention project complete code learning
- Preliminary exploration of basic knowledge of MySQL
- Pytoch loads the initialization V3 pre training model and reports an error
- ZABBIX customized monitoring disk IO performance
- Pytoch through datasets Imagefolder loads datasets directly from files
- Redis master-slave configuration and sentinel mode
猜你喜欢
Pytoch monolayer bidirectional_ LSTM implements MNIST and fashionmnist data classification
激动人心!2022开放原子全球开源峰会报名火热开启!
Learn the memory management of JVM 02 - memory allocation of JVM
Why do you always fail in automated tests?
Knowledge representation (KR)
Take you hand in hand to develop a service monitoring component
Detailed structure and code of inception V3
Principle of universal gbase high availability synchronization tool in Nanjing University
JDBC -- use JDBC connection to operate MySQL database
Take you two minutes to quickly master the route and navigation of flutter
随机推荐
CVPR 2022 | 基于稀疏 Transformer 的单步三维目标识别器
Implementing Yang Hui triangle with cyclic queue C language
Summary of C language learning problems (VS)
MySQL index - extended data
Kotlin变量
About cache exceptions: solutions for cache avalanche, breakdown, and penetration
Flume common commands and basic operations
Solve the problem of cache and database double write data consistency
Deep discussion on the decoding of sent protocol
Differences between IPv6 and IPv4 three departments including the office of network information technology promote IPv6 scale deployment
Learn the memory management of JVM 03 - Method area and meta space of JVM
Anaconda creates a virtual environment and installs pytorch
Migrate data from Mysql to neo4j database
C language structure is initialized as a function parameter
Swift - enables textview to be highly adaptive
MySQL installation, Windows version
ZABBIX agent2 monitors mongodb templates and configuration operations
Solve the error 1045 of Navicat creating local connection -access denied for user [email protected] (using password
POJ-2499 Binary Tree
SAP 自开发记录用户登录日志等信息