Natural language processing Jieba

brief introduction

jieba Is currently performing relatively well Python Chinese word segmentation component .

It mainly supports four word segmentation modes ：

1. Accurate model
2. All model
3. Search engine model
4. paddle Pattern

Support traditional participle 、 Support for custom dictionaries 、MIT License agreement

install

pip install jieba

import jieba
import jieba.posseg as pseg

Precise pattern segmentation

Try to cut the sentence as precisely as possible , Suitable for text analysis ,cut_all Default False

content = " Every month, the Secretary of the industry and Information Technology Bureau has to explain himself to the subordinate departments 24 Installation of technical devices such as oral exchange machine "
print(jieba.cut(content, cut_all=False))
#  To return to the list directly ,  Use jieba.lcut that will do 
print(jieba.lcut(content, cut_all=False))

Running results

<generator object Tokenizer.cut at 0x000001D2FD21D970>
[' Information Technology Department ', ' Female executive ', ' monthly ', ' after ', ' subordinate ', ' department ', ' all ', ' want ', ' In person ', ' Account for ', '24', ' mouth ', ' Switch ', ' etc. ', ' technical ', ' device ', ' Of ', ' install ', ' Work ']

Full mode participle

Scan the sentences for all the words that can be made into words , Very fast , But it can't eliminate ambiguity

content = " Every month, the Secretary of the industry and Information Technology Bureau has to explain himself to the subordinate departments 24 Installation of technical devices such as oral exchange machine "
print(jieba.cut(content, cut_all=True))
#  To return to the list directly ,  Use jieba.lcut that will do 
print(jieba.lcut(content, cut_all=True))

Running results

<generator object Tokenizer.cut at 0x000001D2FD21D970>
[' Information Technology Department ', ' virgin ', ' Female executive ', ' An officer ', ' monthly ', ' menstruation ', ' after ', ' subordinate ', ' department ', ' all ', ' want ', ' In person ', ' Oral sex ', ' Account for ', '24', ' Oral sex ', ' In exchange for ', ' Switch ', ' Replacement ', ' etc. ', ' technology ', ' technical ', ' Sexual organ ', ' device ', ' Of ', ' install ', ' Installer ', ' Tooling ', ' Work ']

Search engine pattern participle

On the basis of exact patterns , Again shred long words , Increase recall rate , Suitable for search engine segmentation

content = " Every month, the Secretary of the industry and Information Technology Bureau has to explain himself to the subordinate departments 24 Installation of technical devices such as oral exchange machine "
print(jieba.lcut_for_search(content))
content = " Trouble is Bodhi , I won't mention "
print(jieba.lcut(content))

Running results

[' Information Technology Department ', ' An officer ', ' Female executive ', ' monthly ', ' after ', ' subordinate ', ' department ', ' all ', ' want ', ' In person ', ' Account for ', '24', ' mouth ', ' In exchange for ', ' Replacement ', ' Switch ', ' etc. ', ' technology ', ' technical ', ' device ', ' Of ', ' install ', ' Work ']
[' Worry ', ' namely ', ' yes ', ' Bodhi ', ',', ' I ', ' For the time being ', ' No ', ' carry ']

Custom dictionary

Custom content writing .userdict.txt in

 Cloud computing  5 n
 Li Xiaofu  2 nr
easy_install 3 eng
 To use  300
 Han Yu appreciates  3 nz
 Bayi Shuanglu  3 nz

print(jieba.lcut(" Bayi Shuanglu changed its name to Bayi Nanchang basketball team ！"))
jieba.load_userdict("./userdict.txt")
print(jieba.lcut(" Bayi Shuanglu changed its name to Bayi Nanchang basketball team ！"))

Running results

[' 8、 ... and ', ' A pair of ', ' deer ', ' renamed ', ' by ', ' Bayi ', ' nanchang ', ' The basketball team ', '！']
[' Bayi Shuanglu ', ' renamed ', ' by ', ' Bayi ', ' nanchang ', ' The basketball team ', '！']

Part of speech tagging

r: Personal pronouns
v: Verb
n: Noun
vn: Gerund

print(pseg.lcut(' I love tian 'anmen square in Beijing '))

Running results

[pair(' I ', 'r'), pair(' Love ', 'v'), pair(' Beijing ', 'ns'), pair(' The tiananmen square ', 'ns')]