Using Jieba and pyhanlp tools to extract keyword words and sentences

Keyword extraction        

         In short, keyword extraction is to extract the words or sentences that best reflect the overall idea from a text . Keywords can help us quickly understand what the text wants to express , Especially in the long literature 、 composition 、 Patents are huge 、 The scene with most content can play a good effect .

        Keyword extraction has the following methods ：TF-IDF、TextRank, The tools are as follows ：jieba、pyhanlp

        About jieba and pyhanlp The implementation of word segmentation can be seen in this blog ： Knowledge map — jieba、pyhanlp、smoothnlp Tools to achieve Chinese word segmentation （ Part of speech table ）

Contents of this article

One 、TF-IDF Algorithm

        1.1、TF-IDF Algorithm is introduced

        1.2、TF-IDF Algorithm calculation

                1） Calculate word frequency

                2） Calculate the inverse document frequency

                3） Calculation TF-IDF

Two 、TextRank Algorithm

        2.1、TextRank Algorithm is introduced

        2.2、TextRank Algorithm calculation

3、 ... and 、 Use sklearn Realization TF-IDF Algorithm

Four 、 Use jiaba Realization TF-IDF Algorithm

5、 ... and 、pyhanlp Realization TextRank Algorithm

        5.1、 Keyword extraction

        5.2、 Keyword extraction

        5.3、 Key sentence extraction

6、 ... and 、jieba Realization TextRank Algorithm

---

One 、TF-IDF Algorithm

        1.1、TF-IDF Algorithm is introduced

                TF-IDF（term frequency–inverse document frequency） It is a commonly used weighting technology for information retrieval and data mining .TF It's word frequency (Term Frequency),IDF It's the inverse text frequency index (Inverse Document Frequency).TF-IDF It's a statistical method , Used to evaluate the importance of words or sentences in a text set or document . The importance of a word or sentence increases in direct proportion to the number of times it appears in the document , At the same time, it will decrease inversely with its frequency in the total document .

                TF-IDF The main idea of ： If the frequency of a word or phrase in an article TF high , And it's rarely seen in other articles , It is believed that this word or phrase has a good ability of classification , Suitable for classification .TF-IDF It's actually ：TF * IDF,TF Word frequency (Term Frequency),IDF Reverse file frequency (Inverse Document Frequency).TF Indicates that the entry is in the document d Frequency of occurrence in . Word frequency (term frequency,TF） It refers to the frequency of a given word appearing in the file . This number is the number of words (term count) Normalization of , To prevent it from leaning towards long documents . Reverse file frequency （inverse document frequency,IDF） Is a measure of the universal importance of words . Of a particular word IDF, You can divide the total number of files by the number of files containing the word , Then take the logarithm of the obtained quotient to get .

                 The high frequency of words in a particular document , And the low file frequency of the word in the entire file set , Can produce high weight TF-IDF. therefore ,TF-IDF Tends to filter out common words , Keep important words .

        1.2、TF-IDF Algorithm calculation

                1） Calculate word frequency

​

                2） Calculate the inverse document frequency

​

                3） Calculation TF-IDF

​

Two 、TextRank Algorithm

        2.1、TextRank Algorithm is introduced

                TextRank The basic idea of the algorithm is to treat documents as a network of words , The links in the network represent the semantic relationship between words .TextRank The algorithm itself is based on Google pagerank The idea of Algorithm . It uses the co-occurrence information between words in a document ( semantics ) Extracting keywords , It can extract the keywords of a given text 、 Key phrases 、 Key sentences .

        2.2、TextRank Algorithm calculation

                ​

3、 ... and 、 Use sklearn Realization TF-IDF Algorithm

from sklearn.feature_extraction.text import TfidfTransformer,CountVectorizer


text = [
        "TF-IDF The main idea of ： If the frequency of a word or phrase in an article TF high , And it's rarely seen in other articles , It is believed that this word or phrase has a good ability of classification , Suitable for classification ."
        ]

#  Word vector model 
vector_model = CountVectorizer()
# tf-idf Algorithm 
tfidf_model = TfidfTransformer()

#　 The amount of text turning 
text_vector = vector_model.fit_transform(text)

#  Extract words from text 
word = vector_model.get_feature_names()

#  Calculation ti-idf value 
tfidf_count = tfidf_model.fit_transform(text_vector)
#  Corresponding ti-idf The weight 
tfidf_weight = tfidf_count.toarray()

for i in range(len(tfidf_weight)):         # Of each book tf-idf Word weight 
    print(list(zip(word,tfidf_weight[i])))
    # print(list([word, weight[i]]))

'''

[('idf The main idea of ', 0.4082482904638631), ('tf', 0.4082482904638631), (' It is believed that this word or phrase has a good ability of classification ', 0.4082482904638631), (' If the frequency of a word or phrase in an article tf high ', 0.4082482904638631), (' And it's rarely seen in other articles ', 0.4082482904638631), (' Suitable for classification ', 0.4082482904638631)]

'''

Four 、 Use jiaba Realization TF-IDF Algorithm

from jieba import analyse


def jieba_tfidf(text):

    # jiba Self contained idf Word frequency file （xxxxx\Lib\site-packages\jieba\analyse\idf.txt）
    tfidf = analyse.extract_tags

    keywords = tfidf(text)
    print(keywords)

    '''
        ['TF', ' The phrase ', ' article ', 'IDF', ' This word ', ' Category ', ' appear ', ' A piece of ', ' Some ', ' distinguish ', ' frequency ', ' fit ', ' classification ', ' be used for ', ' Very few ', ' thought ', ' also ', ' perhaps ', ' Ability ', ' have ']
    '''


if __name__ == '__main__':
    text = '''
            TF-IDF The main idea of ： If the frequency of a word or phrase in an article TF high , And it's rarely seen in other articles , It is believed that this word or phrase has a good ability of classification , Suitable for classification 
    '''

    jieba_tfidf(text)

5、 ... and 、pyhanlp Realization TextRank Algorithm

        

        5.1、 Keyword extraction

from pyhanlp import *


def demo_keyword(content):
    ''' Keyword extraction '''

    TextRankKeyword = JClass("com.hankcs.hanlp.summary.TextRankKeyword")
    #                                     Text     Return to the top topN Number of keywords 
    keyword_list = HanLP.extractKeyword(content, 5)
    print(keyword_list)
        
    '''
        [ Instant noodles ,  right ,  eat ,  high-speed rail ,  moral ]
    '''


if __name__ == "__main__":

    content = (
    """ In recent days, , A video of a man eating instant noodles on a high-speed railway and being angrily resented by a female passenger triggered a heated debate . The woman is emotional , The words are fierce , Scold the passenger loudly , It is said that there is a regulation on the high-speed railway that instant noodles are not allowed , Question its “ Do you have a public conscience ”“ Bad manner ”. After video exposure , The woman responded , Because their children are allergic to instant noodles , Once communicated with this man , But the other party insisted on not listening , She just vented her dissatisfaction , And said that the man's video upload has violated her privacy and reputation , Will take legal measures .12306 The customer service staff said , high-speed rail 、 Instant noodles are generally not sold on bullet trains , But there is no high-speed rail 、 You can't eat instant noodles on the bullet train .
      High speed railway belongs to the space with strong sealing , Every passenger is responsible for maintaining order in the high-speed railway , Obligation not to damage the air quality in the space . This is also the basic quality that passengers should have as citizens . however , Under the background that the high-speed railway does not explicitly prohibit the consumption of instant noodles and other foods , Obstruct others from eating certain food on the grounds of influencing themselves or their children and scold them loudly , I'm afraid it's beyond the right boundary . When people are active in public places , It is not advisable to interfere excessively with the rights of others , Only in this way can we build a harmonious and beautiful public order .
      Generally speaking , The rights of individuals are the obligations of others , No one is allowed to infringe the rights of others at will , This is how every citizen can work normally 、 The basic conditions of life . If rights can be wantonly violated without relief , Society will not work , People also have no happiness . As the Western proverb says ,“ Your right ends at the tip of my nose ”,“ You can sing , But I can't destroy my dream at midnight ”. Whatever rights , The premise that it can be exercised is not to affect the normal life of others , Do not violate public interests and public order and good customs . Beyond this boundary , Rights are no longer rights , Will no longer be protected .
      stay “ Men's high-speed rail were angry when they ate instant noodles ” Incident , First look , Men who eat instant noodles may violate public order , It's your fault to be angry , Not really . Although the high-speed railway is a closed space , But it is different from the subway, which prohibits the consumption of irritating food , Although the high-speed rail operator does not recommend eating spicy food such as instant noodles , But there is no prohibition . thus it can be seen , Even if eating instant noodles 、 durian 、 Spicy hot and other foods may produce pungent taste , Make others uncomfortable , But whether to eat this food , It still depends on personal preferences , Others have no right to interfere or even scold at will . This is also after the disclosure of this event , Many netizens did not criticize the man who ate instant noodles , Instead, I think female passengers should not be loud .
      modern society , The obligations of citizens are generally divided into legal obligations and moral obligations . If an act is defined as a legal obligation , The actor must abide by , In case of violation of , Whether it's the victim or the bystander , Have the right to stop 、 complaints 、 report . Offenders will be punished as they deserve , Will also be morally condemned , Those who actively stop are encouraged to do righteous deeds . If someone violates a moral obligation , Should be condemned by morality and public opinion , And may be investigated for legal responsibility . Such as spitting in public places 、 drop litter carelessly 、 take off your shoes 、 Jump in line at will . here , If the perpetrator ignores the dissuasion of others or even takes revenge , There is no doubt that they will be severely punished .
      Of course , With the development of society , Some moral obligations may rise to legal obligations . As before , Many people don't think much of smoking in public places , Smokers can smoke like no one else . Now? , If someone still smokes in public places without knowing the time , Will inevitably become the target of public criticism .
      Back to “ High speed railway eats instant noodles ” event , If with the renewal of people's ideas , It has gradually become a consensus that you should not eat instant noodles and other foods that may produce irritating smell on the high-speed rail , Or rise to moral or legal obligations . Rebuke 、 It will be justifiable to stop others from eating instant noodles , Otherwise, it's hard to get rid of “ Affectation ”,“ Put self rights above the rights of others ” The suspicion of .
      Under the background that relevant departments have not banned eating instant noodles on high-speed rail , Whether to eat instant noodles is a personal right or personal virtue , It is a part of personal normal life that does not violate the public interest . If you think that others' eating instant noodles makes you uncomfortable , It's best to ask others to cooperate and thank them , Instead of forcibly intervening from the moral vantage point . Only when everyone exercises his rights, he does not cross the border , Speak well when communicating with others , Don't put happiness and comfort above others too selfishly , People tend to be more equal , Public life is more beautiful and orderly ."""
    )

    demo_keyword(content)

        

        5.2、 Keyword extraction

from pyhanlp import *


def demo_phrase_extractor(text):
    """  Phrase extraction  """
    TextRankPy = JClass("com.hankcs.hanlp.summary.TextRankSentence")

    #                                       Return the number 
    phrase_list = HanLP.extractPhrase(text, 5)
    print(phrase_list)

    '''
        [ Eat instant noodles ,  Eat instant noodles ,  moral obligation ,  Fury ,  Instant noodles man ]
    '''


if __name__ == "__main__":


    content = (
        """ In recent days, , A video of a man eating instant noodles on a high-speed railway and being angrily resented by a female passenger triggered a heated debate . The woman is emotional , The words are fierce , Scold the passenger loudly , It is said that there is a regulation on the high-speed railway that instant noodles are not allowed , Question its “ Do you have a public conscience ”“ Bad manner ”. After video exposure , The woman responded , Because their children are allergic to instant noodles , Once communicated with this man , But the other party insisted on not listening , She just vented her dissatisfaction , And said that the man's video upload has violated her privacy and reputation , Will take legal measures .12306 The customer service staff said , high-speed rail 、 Instant noodles are generally not sold on bullet trains , But there is no high-speed rail 、 You can't eat instant noodles on the bullet train .
          High speed railway belongs to the space with strong sealing , Every passenger is responsible for maintaining order in the high-speed railway , Obligation not to damage the air quality in the space . This is also the basic quality that passengers should have as citizens . however , Under the background that the high-speed railway does not explicitly prohibit the consumption of instant noodles and other foods , Obstruct others from eating certain food on the grounds of influencing themselves or their children and scold them loudly , I'm afraid it's beyond the right boundary . When people are active in public places , It is not advisable to interfere excessively with the rights of others , Only in this way can we build a harmonious and beautiful public order .
          Generally speaking , The rights of individuals are the obligations of others , No one is allowed to infringe the rights of others at will , This is how every citizen can work normally 、 The basic conditions of life . If rights can be wantonly violated without relief , Society will not work , People also have no happiness . As the Western proverb says ,“ Your right ends at the tip of my nose ”,“ You can sing , But I can't destroy my dream at midnight ”. Whatever rights , The premise that it can be exercised is not to affect the normal life of others , Do not violate public interests and public order and good customs . Beyond this boundary , Rights are no longer rights , Will no longer be protected .
          stay “ Men's high-speed rail were angry when they ate instant noodles ” Incident , First look , Men who eat instant noodles may violate public order , It's your fault to be angry , Not really . Although the high-speed railway is a closed space , But it is different from the subway, which prohibits the consumption of irritating food , Although the high-speed rail operator does not recommend eating spicy food such as instant noodles , But there is no prohibition . thus it can be seen , Even if eating instant noodles 、 durian 、 Spicy hot and other foods may produce pungent taste , Make others uncomfortable , But whether to eat this food , It still depends on personal preferences , Others have no right to interfere or even scold at will . This is also after the disclosure of this event , Many netizens did not criticize the man who ate instant noodles , Instead, I think female passengers should not be loud .
          modern society , The obligations of citizens are generally divided into legal obligations and moral obligations . If an act is defined as a legal obligation , The actor must abide by , In case of violation of , Whether it's the victim or the bystander , Have the right to stop 、 complaints 、 report . Offenders will be punished as they deserve , Will also be morally condemned , Those who actively stop are encouraged to do righteous deeds . If someone violates a moral obligation , Should be condemned by morality and public opinion , And may be investigated for legal responsibility . Such as spitting in public places 、 drop litter carelessly 、 take off your shoes 、 Jump in line at will . here , If the perpetrator ignores the dissuasion of others or even takes revenge , There is no doubt that they will be severely punished .
          Of course , With the development of society , Some moral obligations may rise to legal obligations . As before , Many people don't think much of smoking in public places , Smokers can smoke like no one else . Now? , If someone still smokes in public places without knowing the time , Will inevitably become the target of public criticism .
          Back to “ High speed railway eats instant noodles ” event , If with the renewal of people's ideas , It has gradually become a consensus that you should not eat instant noodles and other foods that may produce irritating smell on the high-speed rail , Or rise to moral or legal obligations . Rebuke 、 It will be justifiable to stop others from eating instant noodles , Otherwise, it's hard to get rid of “ Affectation ”,“ Put self rights above the rights of others ” The suspicion of .
          Under the background that relevant departments have not banned eating instant noodles on high-speed rail , Whether to eat instant noodles is a personal right or personal virtue , It is a part of personal normal life that does not violate the public interest . If you think that others' eating instant noodles makes you uncomfortable , It's best to ask others to cooperate and thank them , Instead of forcibly intervening from the moral vantage point . Only when everyone exercises his rights, he does not cross the border , Speak well when communicating with others , Don't put happiness and comfort above others too selfishly , People tend to be more equal , Public life is more beautiful and orderly ."""
    )


    demo_phrase_extractor(content)

        5.3、 Key sentence extraction

# encoding=utf-8

from pyhanlp import *


def sentence_extract(content):
    '''  Key sentence extraction  '''
    TextRankSentence = JClass("com.hankcs.hanlp.summary.TextRankSentence")

    #                                             Return the number 
    sentence_list = HanLP.extractSummary(content, 3)
    print(sentence_list)

    
    '''
        [ Men's high-speed rail were angry when they ate instant noodles ,  Men who eat instant noodles may violate public order ,  High speed railway eats instant noodles ]
    '''

if __name__ == '__main__':
    """ Automatic summarization """
    content = (
        """ In recent days, , A video of a man eating instant noodles on a high-speed railway and being angrily resented by a female passenger triggered a heated debate . The woman is emotional , The words are fierce , Scold the passenger loudly , It is said that there is a regulation on the high-speed railway that instant noodles are not allowed , Question its “ Do you have a public conscience ”“ Bad manner ”. After video exposure , The woman responded , Because their children are allergic to instant noodles , Once communicated with this man , But the other party insisted on not listening , She just vented her dissatisfaction , And said that the man's video upload has violated her privacy and reputation , Will take legal measures .12306 The customer service staff said , high-speed rail 、 Instant noodles are generally not sold on bullet trains , But there is no high-speed rail 、 You can't eat instant noodles on the bullet train .
          High speed railway belongs to the space with strong sealing , Every passenger is responsible for maintaining order in the high-speed railway , Obligation not to damage the air quality in the space . This is also the basic quality that passengers should have as citizens . however , Under the background that the high-speed railway does not explicitly prohibit the consumption of instant noodles and other foods , Obstruct others from eating certain food on the grounds of influencing themselves or their children and scold them loudly , I'm afraid it's beyond the right boundary . When people are active in public places , It is not advisable to interfere excessively with the rights of others , Only in this way can we build a harmonious and beautiful public order .
          Generally speaking , The rights of individuals are the obligations of others , No one is allowed to infringe the rights of others at will , This is how every citizen can work normally 、 The basic conditions of life . If rights can be wantonly violated without relief , Society will not work , People also have no happiness . As the Western proverb says ,“ Your right ends at the tip of my nose ”,“ You can sing , But I can't destroy my dream at midnight ”. Whatever rights , The premise that it can be exercised is not to affect the normal life of others , Do not violate public interests and public order and good customs . Beyond this boundary , Rights are no longer rights , Will no longer be protected .
          stay “ Men's high-speed rail were angry when they ate instant noodles ” Incident , First look , Men who eat instant noodles may violate public order , It's your fault to be angry , Not really . Although the high-speed railway is a closed space , But it is different from the subway, which prohibits the consumption of irritating food , Although the high-speed rail operator does not recommend eating spicy food such as instant noodles , But there is no prohibition . thus it can be seen , Even if eating instant noodles 、 durian 、 Spicy hot and other foods may produce pungent taste , Make others uncomfortable , But whether to eat this food , It still depends on personal preferences , Others have no right to interfere or even scold at will . This is also after the disclosure of this event , Many netizens did not criticize the man who ate instant noodles , Instead, I think female passengers should not be loud .
          modern society , The obligations of citizens are generally divided into legal obligations and moral obligations . If an act is defined as a legal obligation , The actor must abide by , In case of violation of , Whether it's the victim or the bystander , Have the right to stop 、 complaints 、 report . Offenders will be punished as they deserve , Will also be morally condemned , Those who actively stop are encouraged to do righteous deeds . If someone violates a moral obligation , Should be condemned by morality and public opinion , And may be investigated for legal responsibility . Such as spitting in public places 、 drop litter carelessly 、 take off your shoes 、 Jump in line at will . here , If the perpetrator ignores the dissuasion of others or even takes revenge , There is no doubt that they will be severely punished .
          Of course , With the development of society , Some moral obligations may rise to legal obligations . As before , Many people don't think much of smoking in public places , Smokers can smoke like no one else . Now? , If someone still smokes in public places without knowing the time , Will inevitably become the target of public criticism .
          Back to “ High speed railway eats instant noodles ” event , If with the renewal of people's ideas , It has gradually become a consensus that you should not eat instant noodles and other foods that may produce irritating smell on the high-speed rail , Or rise to moral or legal obligations . Rebuke 、 It will be justifiable to stop others from eating instant noodles , Otherwise, it's hard to get rid of “ Affectation ”,“ Put self rights above the rights of others ” The suspicion of .
          Under the background that relevant departments have not banned eating instant noodles on high-speed rail , Whether to eat instant noodles is a personal right or personal virtue , It is a part of personal normal life that does not violate the public interest . If you think that others' eating instant noodles makes you uncomfortable , It's best to ask others to cooperate and thank them , Instead of forcibly intervening from the moral vantage point . Only when everyone exercises his rights, he does not cross the border , Speak well when communicating with others , Don't put happiness and comfort above others too selfishly , People tend to be more equal , Public life is more beautiful and orderly ."""
    )

    sentence_extract(content)

6、 ... and 、jieba Realization TextRank Algorithm

'''
    textrank
'''
from jieba import analyse

def jieba_textrank(text):
    textrank = analyse.textrank  #  introduce jieba Medium TextRank
    keywords = textrank(text, 5)
    print(keywords)

if __name__ == '__main__':
    text = """
         In recent days, , A video of a man eating instant noodles on a high-speed railway and being angrily resented by a female passenger triggered a heated debate . The woman is emotional , The words are fierce , Scold the passenger loudly , It is said that there is a regulation on the high-speed railway that instant noodles are not allowed , Question its “ Do you have a public conscience ”“ Bad manner ”. After video exposure , The woman responded , Because their children are allergic to instant noodles , Once communicated with this man , But the other party insisted on not listening , She just vented her dissatisfaction , And said that the man's video upload has violated her privacy and reputation , Will take legal measures .12306 The customer service staff said , high-speed rail 、 Instant noodles are generally not sold on bullet trains , But there is no high-speed rail 、 You can't eat instant noodles on the bullet train .
          High speed railway belongs to the space with strong sealing , Every passenger is responsible for maintaining order in the high-speed railway , Obligation not to damage the air quality in the space . This is also the basic quality that passengers should have as citizens . however , Under the background that the high-speed railway does not explicitly prohibit the consumption of instant noodles and other foods , Obstruct others from eating certain food on the grounds of influencing themselves or their children and scold them loudly , I'm afraid it's beyond the right boundary . When people are active in public places , It is not advisable to interfere excessively with the rights of others , Only in this way can we build a harmonious and beautiful public order .
          Generally speaking , The rights of individuals are the obligations of others , No one is allowed to infringe the rights of others at will , This is how every citizen can work normally 、 The basic conditions of life . If rights can be wantonly violated without relief , Society will not work , People also have no happiness . As the Western proverb says ,“ Your right ends at the tip of my nose ”,“ You can sing , But I can't destroy my dream at midnight ”. Whatever rights , The premise that it can be exercised is not to affect the normal life of others , Do not violate public interests and public order and good customs . Beyond this boundary , Rights are no longer rights , Will no longer be protected .
          stay “ Men's high-speed rail were angry when they ate instant noodles ” Incident , First look , Men who eat instant noodles may violate public order , It's your fault to be angry , Not really . Although the high-speed railway is a closed space , But it is different from the subway, which prohibits the consumption of irritating food , Although the high-speed rail operator does not recommend eating spicy food such as instant noodles , But there is no prohibition . thus it can be seen , Even if eating instant noodles 、 durian 、 Spicy hot and other foods may produce pungent taste , Make others uncomfortable , But whether to eat this food , It still depends on personal preferences , Others have no right to interfere or even scold at will . This is also after the disclosure of this event , Many netizens did not criticize the man who ate instant noodles , Instead, I think female passengers should not be loud .
          modern society , The obligations of citizens are generally divided into legal obligations and moral obligations . If an act is defined as a legal obligation , The actor must abide by , In case of violation of , Whether it's the victim or the bystander , Have the right to stop 、 complaints 、 report . Offenders will be punished as they deserve , Will also be morally condemned , Those who actively stop are encouraged to do righteous deeds . If someone violates a moral obligation , Should be condemned by morality and public opinion , And may be investigated for legal responsibility . Such as spitting in public places 、 drop litter carelessly 、 take off your shoes 、 Jump in line at will . here , If the perpetrator ignores the dissuasion of others or even takes revenge , There is no doubt that they will be severely punished .
          Of course , With the development of society , Some moral obligations may rise to legal obligations . As before , Many people don't think much of smoking in public places , Smokers can smoke like no one else . Now? , If someone still smokes in public places without knowing the time , Will inevitably become the target of public criticism .
          Back to “ High speed railway eats instant noodles ” event , If with the renewal of people's ideas , It has gradually become a consensus that you should not eat instant noodles and other foods that may produce irritating smell on the high-speed rail , Or rise to moral or legal obligations . Rebuke 、 It will be justifiable to stop others from eating instant noodles , Otherwise, it's hard to get rid of “ Affectation ”,“ Put self rights above the rights of others ” The suspicion of .
          Under the background that relevant departments have not banned eating instant noodles on high-speed rail , Whether to eat instant noodles is a personal right or personal virtue , It is a part of personal normal life that does not violate the public interest . If you think that others' eating instant noodles makes you uncomfortable , It's best to ask others to cooperate and thank them , Instead of forcibly intervening from the moral vantage point . Only when everyone exercises his rights, he does not cross the border , Speak well when communicating with others , Don't put happiness and comfort above others too selfishly , People tend to be more equal , Public life is more beautiful and orderly .
    """

    jieba_textrank(text)