Stutter participle_ Principle of word breaker

Hello everyone , I'm wang Jun , An architect who can write code and recite poetry . Today, let's talk about the stutter participle _ Principle of word splitter , I hope it can help you make progress !!!

install jieba library ：pip3 install jieba

# Stuttering participle 
# -*- coding:utf-8 -*-
import sys
import os
import jieba

Copy

sent = ' Tianshan intelligence is a business intelligence enterprise BI、 Data analysis 、 Technical community in the field of data mining and big data technology  www.hellobi.com . Content from the initial business intelligence  BI  The field has also been extended to data analysis 、 Data mining is related to big data   In the field of technology , Include  R、Python、SPSS、Hadoop、Spark、Hive、Kylin etc. , Become a vertical community focused on the data field . Tianshan intelligence is committed to building an ecosystem based on the data field , Link everything through the community   Data related resources : For example, the data itself 、 people 、 Data solution providers and enterprises , Work together with everyone to promote big data 、 business intelligence BI Popularization and development in China .'
print (sent)

Copy

Stuttering word segmentation module has three word segmentation modes ：

1. All model ： Scan all the words that can be made into words in a sentence , Very fast , But it doesn't solve the ambiguity . This full mode , According to the dictionary , Match and divide all the words that appear , So there will be repetition , obviously , This is not what we need .

2. Accurate model ： Try to cut the sentence as precisely as possible , Suitable for text analysis （ similar LTP Word segmentation ）, And this precise model is closer to what we want .

3. Search engine model ： Segmentation of long words based on precise patterns , Increase recall rate , Suitable for search engine segmentation . This search engine model is also good , More detailed .

# All model 
wordlist = jieba.cut(sent,cut_all = True)
print('|'.join(wordlist))

Copy

# Exact segmentation 
wordlist = jieba.cut(sent)
print('|'.join(wordlist))  

Copy

# Search engine model 
wordlist = jieba.cut_for_search(sent)
print('|'.join(wordlist))

Copy

Find new problems -- Add user-defined dictionary ： Looking back at the results of the exact model , Find some new words or professional words , for example ： Tianshan intelligence 、 big data , These should no longer be cut apart , So based on the default dictionary , We can load custom dictionaries . Enter my jieba Module directory -> See a dict The dictionary of , open -> Found to have 1. word 2. Numbers （ For word frequency , The higher the height, the easier it is to match ） 3. The part of speech . For convenience , We define and add a dictionary named userdict.txt

# Add user-defined dictionary 

# Use the user dictionary 
jieba.load_userdict('D:\\Anaconda3\\Lib\\site-packages\\jieba\\userdict.txt')
wordlist = jieba.cut(sent)
print('|'.join(wordlist)) 

Copy

Reference material ：

https://zhuanlan.zhihu.com/p/29747350?utm_source=qq&utm_medium=social&utm_oi=780081763178258432

That's the end of today's article , Thank you for reading ,Java Architects must see I wish you a promotion and a raise , Good luck every year .