NLP word segmentation

participle

What is participle

The participle is ⾃ But the language ⾔ understand NLP An important step . A participle is a sentence ⼦、 The paragraph 、⽂ This chapter ⻓⽂ Ben , Decompose into data structures in terms of words ,⽅ For subsequent processing and analysis ⼯ do .

The function of participle

1、 Turn complex problems into mathematical problems

The reason why machine learning seems to solve many complex problems , Because it turns these problems into mathematical problems .⽽ NLP The same idea ,⽂ Ben is ⼀ some 「⾮ Structured data 」, We need to convert these data into 「 Structured data 」, Structured data can be transformed into mathematical problems ,⽽ Word segmentation is the first step of transformation .

2、 Word is a more appropriate granularity

Words are the most important means of expressing complete meaning ⼩ Company . The granularity of words is too ⼩,⽆ Express the full meaning of ,⽐ Such as ”⿏“ It can be ”⽼⿏“, It can also be ”⿏ mark “.⽽ sentence ⼦ The granularity of is too ⼤, Carry more information , It's hard to answer ⽤.⽐ Such as ” Tradition ⽅ The law requires word segmentation ,⼀ An important reason is tradition ⽅ The modeling of long-distance dependence can ⼒ Weak .”

Differences between Chinese and English word segmentation

1. There are different ways of word segmentation , Chinese is more difficult

Britain ⽂ There are natural spaces as separators , But in ⽂ No, . So how to segment is ⼀ A difficulty , Plus medium ⽂⾥⼀ The situation of polysemy ⾮ Often , It's easy to have ambiguity .

2. There are many forms of English words

Britain ⽂ There are abundant deformations and transformations in words . In response to these complex transformations , Britain ⽂NLP phase ⽐ in ⽂ There is ⼀ Some unique processing steps , We call it morphological reduction （Lemmatization） And words ⼲ extract （Stemming）; in ⽂ You don't need to .
Part of speech reduction ：does,done,doing,did It needs to be restored to do.
word ⼲ extract ：cities,children,teeth These words , Need to be converted to city,child,tooth These basic forms .

3. Chinese word segmentation needs to consider granularity

for example 「 China's science and Technology ⼤ learn 」 There are many ways ：
China's science and Technology ⼤ learn
China \ Science and technology \ ⼤ learn
China \ science \ technology \ ⼤ learn
The more granularity ⼤, The more accurate the meaning is , But it can also lead to recalls ⽐ Less . So in ⽂ Different scenarios and requirements are needed to choose different granularity . This is in England ⽂ None of them .

4、 participle ⼯ have