当前位置：网站首页>My seven years with NLP

My seven years with NLP

2022-07-06 06:35:00 【kaiyuan_ sjtu】

author | Pan Haojie
Arrangement | NewBeeNLP

Have a nice weekend ！ It's really hard to finish the first working week of the new year . Today, share a friend with NLP The story of , The article is very interesting , I'm very good x. Read it carefully , You can feel that Yes AI and NLP Blazing faith with bright eyes , I hope you are the same ~

Here is the original .

A few days ago, I discussed with my friend about... In a blind pinch shop in Beijing AR Entrepreneurial ideas and some attempts . On the way home , Talk about whether I have the idea of starting a business , Is there anything you particularly want to do , And talk about 16 An in-depth visual question and answer system for people with visual impairment during the undergraduate period of Zhejiang University in , He said he used this project as a benchmark at that time , I think we have repeatedly participated in various entrepreneurship and innovation competitions for this project from beginning to end , Won a lot of awards , It was already very successful at that time . At this time, I can't help but sweat , Because at that time VQA Technology is still in its infancy , Deep learning is in the early stage of vigorous development , Take it. It looks very “low” Technology everywhere “ try every trick to mislead the public ” I don't think there's much to be proud of .

But the night after the fork at the intersection separated , I can't sleep . The experiences and ideas of the past few years have flooded in , I can't help but think of some ignorant ideas before doing this project , And the jigsaw puzzle in my head . Suddenly looking back , I found that over the years, although the puzzle has become more and more blurred , But when you clean it, it has been spliced one by one , This is undoubtedly the most exciting thing .

Fantasy sprouts

The title says “ Seven years ”, Because I am. 15 In, he was officially exposed to machine learning , Get started AI And NLP Of . But as early as 2012 year , I'm watching a movie called 《 Cherry Blossom Village pet girl 》 New times , I developed it independently for a character at that time, Ryunosuke Akasaka “ Little maid ” There was a great deal of interest , This is an anthropomorphic assistant independently developed by long Zhijie , Can help him deal with some contact matters . Of course, what really attracts me is not this kind of similarity “ Forward mail ” The function of , But some of its anthropomorphic actions 、 The dialogue even likes to long Zhijie and is hostile to other opposite sex .

《 Cherry Blossom Village pet girl 》 Ryunosuke Akasaka appears

I couldn't help thinking , Whether one day , Whether everyone can also have their own second dimension “ partner ”, Can sense human emotions , Be able to talk freely . I Believe , today , This is also why many students enter artificial intelligence , Learn natural language processing （NLP） Your dream .

So after the college entrance examination , With family support , Choose to enter the Engineering Department of Zhejiang University （ Information ） Categories: .

Fantasy impact

Because of the fantasy flame of an animation , If I major in computer science and technology in the future , It's also a little too romantic . At that time, the planning department of Zhejiang University was still in a state of prosperity , The average performance of choosing a major is almost the same as that of automation, which has been ranked first in previous years （ control ） Close , Because I hate chemistry very much , It's just “ By mistake ” Entered the computer major .

stay 2015 I watched 《Her》 The film , It directly prompted me to study the theory and application of artificial intelligence at that time . Affect the subsequent application and guidance of he Zhijun class 、 The choice of postgraduate direction 、 Internship and future job options .

《Her》 What is told is AI System OS1 Avatar Virtual Assistant Samantha（ Marvel widows dubbing ） With the man Theodore（ It's also DC Clown actor ） The love story of . In the movie Samantha Beautiful sound line 、 Having the ability of independent thinking and emotional expression and evolving intelligence have given me a lot of ideological impact , At the same time, I also have a strong longing for such intelligent human-computer interaction .

《Her》 In the movie, the man is installing OS1

I'm still addicted to this fantasy , Imagine that one day machines can resonate emotionally with people . meanwhile , I read a lot of information about artificial intelligence , Understand machine learning , Study Python, During the interview of he Zhijun's class, he repeatedly stressed his research ambition and ideas , I've contacted the people who will have a great impact on me in the future Cai Deng ^[1] teacher .

pattern recognition

2015 year , Like most people at that time , I passed Andrew Ng The teacher's Coursera Introduction to the course machine learning , At the same time, I entered Mr. Caideng's Laboratory （ Zhejiang University CAD&CG Machine learning group ） Explore practical projects , At that time, Miss Cai wanted me to contact an important project in the laboratory at that time “ Automatic license plate recognition ”, But I declined , Because I was just 《Her》 Inspired me to this “ Image recognition ” Not very interested , Bent on doing NLP Direction , Because the laboratory NLP There are not many related projects , He asked me to sit in on the group meeting of the social networking and Data Mining Group .

In this group meeting , I met many excellent students , Some of these seniors later went out for further study （CMU、UIUC etc. ）、 Some have joined industry-leading companies such as apple and Alibaba 、 Some even started a business and got the ten thousand people plan of Zhejiang Province . With their help , I'm getting started with data mining and NLP, And made a comment emotional analysis （Sentiment Analysis） project . Right then AI I don't know much about machine learning , The night before this project, I almost lost sleep excitedly , The reason for insomnia is to conceive your own creation “ Artificial agent ”, I thought , If the machine can communicate with people normally , Many modules are needed in the process , Emotional analysis is naturally very important to judge whether you are happy or unhappy , I even drew a picture in my mind “ Artificial agent ” The flow sketch of the brain , I can't wait to get out of bed coding.

In the study NLP After the relevant foundation , I took it from Kaggle Data downloaded from , Look at all kinds of tutorials , Training models , Predicted results , During this period, I also came into contact with the hot Word2Vec, And trained a number of word vectors by themselves . This project took me some time , I finally finished all kinds of experiments , After self-study Latex Then he took a paper and wrote a research report like a gourd , Delivered to teacher Cai , He was also quite satisfied and agreed to my application for him as an undergraduate tutor .

Although it looks perfect , But there are a few things that disappoint me ： First, I found that my model is completely experimental , If I type in a sentence written by myself , The output is mostly wrong , Second, if I want to use my mother tongue Chinese , The model is impossible to predict , This is different from what I think “ Sentiment analysis ” The machines are very different . At that time, the childish idea actually reflected their worries now , That is, can the model or system obtained by pure data-driven learning really understand our words ？ Does the current method rely too much on data ？ Is this pattern recognition really what we want ？

VQA The little show when China founded the project demo

16 Winter vacation in , I met Pei Shen and Yang leg by chance , Did what the introduction said together VQA project . Now that I think about it , It is a blessing for the college age to meet two awesome teammates. . Pei God has a terrible comprehensive quality , The credits are mine 1~2 But times can always rank first in the Planning Institute , Study 、 An association 、 scientific research 、 Play a game without pulling , stay CMU LTI After graduation, , Currently in Google Cloud Do machine translation ; Foreign legs are earlier than me （ Freshman ） It's in AI field , My research interest is computer vision , stay 14、15 An era when deep learning frameworks were particularly difficult to use , I lost my hand CNN The Internet , At present, I'm strongly recommended to study in Mr. Cai Deng's Laboratory PHD. In the later years of undergraduate , We are almost the most close and smooth “ Golden iron triangle ”.

VQA My idea is that I found one when I was discussing a school level scientific research project with peishen idea, And just then I trained Word Embedding, Found something to do CV My foreign leg , In less than a month Demo Made it out. , And gain guochuang 、 Project initiation of various projects such as Challenge Cup , In the end, he made good achievements in many entrepreneurship and innovation competitions and projects , Did a lot of demo Also applied for a patent .

VQA The project is really what I want to achieve AI A small piece of the blueprint , because 《Her》 Medium Samantha She also needs a camera as her eyes to communicate with her Theodore To communicate , Just by virtue of the technology at that time , We can only reluctantly based on some Pattern Recognition Way to achieve the so-called “ intelligence ”, This was also a point that disappointed me at that time , Because I found a lot of so-called artificial intelligence at that time , It's just simple pattern recognition .

The text generated

Looking back on my last year and a half , Because I have a lot of time to do what I want to do , I really （ To some extent ） Step into the door of intelligent dialogue and human-computer interaction ,17 From the end of the year 18 Early years , Internship experience from Fuxi Laboratory of Netease games to Ali Dharma Academy , From theory to practice and then to academic , From full of hope to disappointment to return to peace ,“ The text generated ” This direction runs through the whole year .

At that time, Fuxi laboratory was still a very small team ,NLP There are only three members , At that time, we explored many directions of text generation , Including resume screening and reason generation 、 Smart desktop generation 、 intelligence NPC When they landed, they ended up unharmed , During the internship , I feel confused more than once . On the other hand , By one of their own “ Eager for a paper ” Dominated by your thoughts , Finally, I began to step into some research work on dialogue generation , At that time in Fuxi and Zhaozhou ^[2] In the cooperation of teachers , Did two things , Namely Dialogue summary （Dialogue Summarization） and Dialog response generation （Dialogue Response Generation）, The former is my first top meeting paper （WWW'19） Of topic, Achieved a breakthrough in scientific research from zero to one , The latter is my bachelor's thesis of Zhejiang University .

The next semester of senior year , I went to Dharma hall on the recommendation of teacher Cai NLP Group （ Si Luo ^[3] Teacher group ）, Research tmall multi review summary （Multi-review Summarization） Topics , This is the first time I have come into contact with the huge scale data of industry , Purify and use text generation technology to create value for users （ Improve the efficiency of users reading comments ）, Finally, it was published in SIGIR Industry Track in （ link ^[4]）.

I remember , When I realize the first Seq2seq Model , First time Memory Network The joy of the paper , And think that under a large amount of dialogue data , use Memory Means to solve the long-term dependence of dialogue , Then based on the dialog generation function , It can really achieve a certain degree of success “ agent ”, Be able to talk to people smoothly . However, as I become more familiar with text generation tasks , I found that it was still far from what I thought , At that time, it was popular to use Memory perhaps Attention Summarize the previous information into a vector , Add to the Seq2seq The task of model generation , This implicit memory does not capture the real intention of user input and its relationship with the previous text , At the same time, it is based on Beam Search At that time, probability generation was often not particularly good in fluency , It also often generates some words that don't match the preface and the Afterword , Lack of logical rigor . Besides , Text generation models are usually used in some highly aligned tasks （ Such as machine translation 、ASR etc. ） Can show a strong advantage , In a dialogue that requires diversity , But often produce the same answer .

This is also part of the original intention of my graduate students, including follow-up work, to solve these problems by means of knowledge .

World knowledge

Me and my mentor Yangqiu Song^[5] The professor met by email , He had just come from WVU To HKUST CSE As an assistant professor , I saw his home page on the Internet , Trying to apply for summer jobs by email RA, Unexpectedly, he readily agreed , Even gave it to me through the Department $10,000 HKD Monthly salary as living expenses in Hong Kong .

What we did was NIST Released a LoReHLT^[6] evaluating , The main research automatically puts one in some rare languages （ Like tigrinian , Oromo , Uyghur, etc ） Extracting scene frame sentences from regional disaster news , And determine whether there are some in the news "issue'' and "need''. Under such a task , I realized for the first time that the heap data training depth model is not omnipotent , Some linguistics is needed , Such as Morphology, And transfer learning, such as Domain Adaptation Importance . Follow me up with Yangqiu After communication , Decided to go HKUST Master's degree and engage in knowledge about the world （World Knowledge） And common sense （Common-sense） Some of our research work , Before going to Hong Kong , My tutor and Dan Roth stay arxiv Published 《Machine learning with world knowledge: The position and survey^[7]》 Carried out translation work , And introduce external knowledge or remote supervision （Distant-Supervision） There are some ideas to assist model learning with knowledge .

ASER 1.0

stay 18 Join in late 2012 HKUST KnowComp After the group , Because I had some conversation related work before , Elder martial brother Hongming^[8] To me and the same Xin^[9] Offer cooperation ASER^[10] Ideas .

ASER（Activities, States, Events, and their Relations） yes Yangqiu One thing I wanted to do when I was in Huawei Noah's Ark , Formerly known as ActivityNet. This is a large-scale event knowledge map , Unlike some knowledge maps on the market at that time , Each node is an event (Eventuality), Not the entity , We hope to find out the relationship between each event element through large-scale data mining , There are already 4.38 Billions of Eventuality and 6.48 The edge of billion .ASER yes HKUST KnowComp It's also Yangqiu One of the Lifelong The job of , I am honored to be one of the initial members , And became comrades in arms with two excellent people in the laboratory .

stay ASER In my first paper WWW' 20 \[ link \]^[11] after , We are still working tirelessly on ASER There are some new thoughts under the prototype system of this first edition , stay 19~21 In, we proposed ASER2.0^[12] , I am right. ASER Made a simple conceptual module , And built a Demo.

ASER 2.0 with Conceptualization

adopt ASER This project , I was surprised to find that common sense knowledge is so charming , And I think background knowledge is definitely a part of my puzzle ,Samantha One of the reasons why we can have such high intelligence is that we keep learning the background knowledge precipitated by human beings for thousands of years , Just know what men and women are in this world 、 Why do you sleep when it's dark .

Eliminate ASER Beyond this memorable project , And my master's thesis also makes me and Yangqiu One who is very excited and forgets to eat and sleep Topic, The source is me AAAI’19 A rejected article is based on Caption and VisualDialogue A piece of Dialogue Summarization The paper ^[13], We were pleasantly surprised to find , Establish the relationship between dialogue and description through pictures , Some deeper knowledge of human linguistics can be excavated , Therefore, we spent a long time studying pragmatics （Pragmatics）, And the corresponding two sub concepts ：

Implicature ：“ Small A Will you attend the party ”“ She has something at home ” → “ Small A I won't attend the party ”
Presupposition ：“ I don't want to go to this restaurant anymore ” → “ I've been to this restaurant once ”

This research is very... For me exciting Of , The example I often give people is “ If your girlfriend sends ‘ I am going to bed ！’” It means she really wants to sleep , Still angry （ This is often encountered in daily life lol）. Of course, our research results are not accepted by most ACL And EMNLP Recognized by linguistic experts , Think we're half baked linguists （ It is ）, Finally, after several rounds of contributions after graduation , Still studying, brother Tianqing^[14] With the help of Dialogue Summarization Published on AKBC'21 ( link ^[15]) On

My experience as a graduate student , Let me know the world in a very comfortable state 、 Have a deeper understanding of common sense and Linguistics , It's definitely a very colorful piece of my puzzle . Of course , Because I was obsessed with the construction of world knowledge , Missed NLP Breakthrough work that is difficult to meet in ten years —— Large scale pre training language model （PLM）.

Industry landing

from 15 From the second half of the year to 19 The second half of the year , Experienced 4 Years of technical polishing 、 Precipitation and to NLP The understanding of the , I'm facing an important turning point in my life , It's direct employment , Keep reading PHD？ This problem also puzzles thousands of students . I dissected myself again and again , I don't think I'm the kind of person who is deeply engaged in scientific research in a specific field , Make products and applications more systematically , Even making the final state of my fantasy can bring me a greater sense of achievement .

Therefore, by trying to learn and establish the reason for the implementation of industrial applications , Plus admiring teacher Jia Yangqing , I joined Alibaba cloud computing platform , stay PAI Development of large-scale transfer learning framework for machine learning , And landing a lot of applications . But the mistake is , Because of the upsurge of pre training language models , We transfer learning /NLP The group turned in this direction , Make me to BERT And then a lot of PLM With a very deep understanding , In addition, it came a long time ago GPT-3 A series of major breakthroughs in text generation , It made me more interested in it .

My work in Ali mainly focuses on EasyTransfer^[16] The development of this framework （ It's open source now ）, Within the group （ Such as Tmall 、 Salted fish 、 youku ） Algorithm students provide large-scale and high-performance migration learning （ Include PLM） Distributed training and prediction , For customers outside the group NLP Customization of product solutions . At the same time, this year is also the year of my thesis blowout , Most of the work left over from the previous student days has been in the draft , In Ali Leader Minghui^[17] with colleagues Chengyu^[18] With the help of, I also published the first article in my life ACL、EMNLP、CIKM,etc.

EasyTransfer Github Home page

A project slightly participated in during his stay in Ali M6 (paper^[19], And the product experience links I made ^[20]) Let me see the powerful ability of large-scale multimodal pre training model , And made a friend Junyang^[21] bosses . So I can't help thinking , Over the years, I have insisted on doing text related work , But whether the text is enough to represent world knowledge , Is it enough for an agent to understand the world , Understand what people say , People's thoughts and complex emotions ？

With such an idea , Plus personal reasons and workplace changes , With Wang Zhongyuan ^[22] Several rounds of communication from the teacher , I came to Beijing , Added Kwai Fu multimodal content understanding department. （MMU）, I hope to be able to Kwai ten billion video and text. , Understand human knowledge , And explore the mode of human communication in the era of short video 、 The new way of existence of world knowledge under the change of knowledge media and the knowledge background required by future agents .

Jigsaw puzzle reorganization

Of course, the reality is not always romantic , I know the ups and downs I have experienced in recent years , Even for a long time, I have forgotten why I studied computer , Why do AI, Why research NLP And world knowledge . Every experience is like a small fragment , Scattered in the depths of memory , But through some opportunities , After reorganizing it , I clearly feel the happiness that the puzzle is gradually improved , You can share with your friends excitedly “ I've been on the road ”. After the picture depicted by the puzzle becomes clearer and clearer , Then faith came into being , The joy of faith is unparalleled . And I'm lucky to meet many people who accompany me on this road , My mentor 、 Teammates and collaborators in various projects 、 My family and friends .

As for the future , I will firmly seize the opportunity of this puzzle reorganization , Gradually realize what you really want to achieve . I've read a lot of news about agents recently , Including virtual human and chat robot , Including metauniverse and artificial intelligence , I'm lucky to be able to , We can do such interesting things together . Although reality and ideal are still far away , I still believe in the rapid evolution of technology in academia and industry , Be able to gradually approach the preliminary world knowledge 、 An agent with its own personality and emotion , And like a black stone 《 Clara and the sun 》 Like a companion robot in , Have a strong curiosity about the world 、 Observation and empathy （ in fact , The industry already has a virtual chat Bots Can do well ）. After that, I will also make some preliminary exploration , If there is a friend who has the same fantasy as me , You can also communicate with me [email protected]

—— Xin Chou Nian The twelfth month is twenty-nine Birth diary

Communicate together

I want to learn and progress with you ！『NewBeeNLP』 At present, many communication groups in different directions have been established （ machine learning / Deep learning / natural language processing / Search recommendations / Figure network / Interview communication / etc. ）, Quota co., LTD. , Quickly add the wechat below to join the discussion and exchange ！（ Pay attention to it o want Notes Can pass ）

Resources for this article

[1]

Cai Deng : https://www.cad.zju.edu.cn/home/dengcai/

[2]

Zhaozhou : https://person.zju.edu.cn/zhaozhou

[3]

Si Luo : https://www.cs.purdue.edu/homes/lsi/

[4]

link : https://dl.acm.org/doi/abs/10.1145/3397271.3401439

[5]

Yangqiu Song: https://www.cse.ust.hk/~yqsong/

[6]

LoReHLT: https://www.nist.gov/itl/iad/mig/lorehlt-evaluations

[7]

Machine learning with world knowledge: The position and survey: https://arxiv.org/abs/1705.02908

[8]

Hongming: https://panda0881.github.io/Hongming_Homepage/

[9]

Xin: https://cse.hkust.edu.hk/~xliucr/

[10]

ASER: https://github.com/HKUST-KnowComp/ASER

[11]

[ link ]: https://dl.acm.org/doi/10.1145/3366423.3380107

[12]

ASER2.0: https://arxiv.org/abs/2104.02137

[13]

The paper : https://arxiv.org/abs/1811.00185

[14]

Tianqing: https://fangtq.com/

[15]

link : https://openreview.net/forum%3Fid%3DAJKd0iIFMDc

[16]

EasyTransfer: https://github.com/alibaba/EasyTransfer

[17]

Minghui: https://sites.google.com/site/qiumh0727

[18]

Chengyu: https://chywang.github.io/

[19]

paper: https://arxiv.org/abs/2104.02137

[20]

Experience Links : https://workbench.data.aliyun.com/experience_new.htm%3Fspm%3Da2c0j.14066474.J_3832552950.1.7b34426a4BMe7x%26lang%3Dzh-CN%23/m6/modules

[21]

Junyang: https://justinlin610.github.io/%23intro

[22]

Wang Zhongyuan : https://www.wangzhongyuan.com/

- END -