ML - natural language processing - Introduction to natural language processing

Reference resources

Huawei cloud College

What is natural language processing

The use of computers as a tool for human specific written and oral forms of natural language information , The technique of carrying out various types of treatment and processing .—— Feng Zhiwei
Natural language processing can be defined as a discipline that studies language problems in human communication and human computer communication . Naturallanguageprocessing needs to develop models that represent language capabilities and language applications , Build a computing framework to implement such a language model , Propose corresponding methods to continuously improve such a language model , According to this language model, various practical systems are designed , And discuss the evaluation technology of these practical systems .——Bill Manaris

Basic methods of natural language processing

（1） Capability model
A model based on linguistic rules . Based on the assumption that there is a general rule of grammar in the human brain , It is believed that language is derived from the language ability of human brain . Building a language model is to simulate this innate language ability by establishing a manually edited language rule set
also called “ idealistic ” Language model , The representatives are Chomsky,Minsky.
Modeling steps ：（1） Formalization of linguistic knowledge .（2） Formalization rule algorithmization .（3） Algorithm implementation .
（2） Application model
Specific language models based on different language processing applications , Learn complex by building specific mathematical models , Extensive language structure , Then use statistics 、 Pattern recognition and machine learning are used to train the parameters of the model , To expand the scale of language use .
also called “ Empirical ” Language model , The representatives are Shannon,Skinner.
Modeling steps ：（1） Obtain statistical information on different levels of language units from a large-scale real corpus .（2） According to the statistical information on the lower level language units, relevant statistical reasoning techniques are used , To calculate the statistics of teaching high-level language units .
stay NLP The development of , Methods can be roughly divided into the following categories ：（1） A rule-based approach .（2） Statistical based methods .

Three levels of natural language processing

Lexical analysis ： Including participles 、 Part of speech tagging 、 Named entity recognition, etc .
Syntactic parsing ： Including syntactic structure analysis and dependency analysis .
Semantic analysis ： The ultimate goal is to understand the true semantics of sentences .

Difficulties in natural language processing

The development status of natural language processing

A number of influential language databases have been developed , Some technologies have reached or basically reached the degree of practicality , And play a great role in practical application . Peking University corpus 、HowNet. There are many new research directions , Such as reading comprehension 、 Images （ video ） understand 、 Simultaneous interpretation of speech .
In addition to some achievements , There are still many theoretical problems that have not been fundamentally solved , For example, the recognition of unlisted words 、 Disambiguation 、 Semantic understanding 、 A complete set is missing 、 Systematic theoretical framework system .