Chapter 2 Introduction to key technologies

   

Catalog

2.1 Python Reptiles

2.2 Python Data analysis  

---

        This project is through Python Language to design and implement . The main technologies used in the project are Python Reptiles ,Python Data analysis, etc . The data comes from www.fangjia.com.

2.1 Python Reptiles

       Python Reptiles It is a common tool for collecting Internet data , In recent years, with the development of the Internet, it has developed rapidly . Using web crawlers to crawl network data, we must first understand the concept and main classification of the network , System structure of various reptiles 、 How it works , Common strategies , And the main application scenarios , meanwhile , For the sake of copyright and data security , We also need to understand the current application legitimacy of crawlers and the agreements that need to be observed when crawling websites . at present , Most websites allow the data crawled by crawlers to be used for personal use or scientific research . But if the crawled data is used for other purposes , Especially reprint or commercial use , Serious will violate the law or cause civil disputes . The following two kinds of data cannot be crawled , Not for commercial use . Personal privacy data ： Such as name 、 Phone number 、 Age 、 Blood type 、 Marital status, etc , Crawling such data will violate the personal information protection law . Data that is explicitly prohibited from being accessed by others ： For example, the user has set permission controls such as account and password , Encrypted content . Attention should also be paid to copyright related issues , Copyrighted content signed by the author is not allowed to be reproduced or used for commercial purposes after crawling .

2.2 Python Data analysis  

        Data analysis refers to the analysis of a large number of collected data with appropriate analytical methods , Extract useful information and form conclusions , The process of studying and summarizing the data in detail . Data mining in broad sense includes data analysis and data mining in narrow sense . In a narrow sense, data analysis refers to data analysis according to the purpose of analysis , Comparative analysis 、 Group analysis 、 Cross analysis and regression analysis , Process and analyze the collected data , Extract valuable information , Play the role of data , The process of obtaining the result of a characteristic statistic . Data mining is from a large number of 、 Not completely 、 Noisy 、 Vague 、 In random practical application data , By applying clustering model 、 Classification model 、 Regression and association rules , The process of tapping potential value . At present, the mainstream data analysis languages are Python、R、Matlab These three . among ,Python Has a rich and powerful library , It's often called glue language , To be able to make various modules in other languages （ In especial c and c++) Easily connected together , It's easier to learn 、 More rigorous programming language .R Language is used for statistical analysis 、 Drawing language and operating environment . It belongs to GNU A freedom of the system 、 free 、 Open source software .Matlab Is used to perform matrix operations 、 Plotting functions and data 、 Implementation algorithm 、 Create user interfaces and programs that connect to other programming languages , It is mainly used in engineering calculation 、 Control design 、 Signal processing and communication 、 The image processing 、 Signal detection 、 Financial modeling, design and analysis .

Python Data analysis mainly includes the following five advantages .

1. The grammar is simple and concise
2. There are many powerful Libraries
3. Powerful
4. Not only for research and prototyping , It also applies to building production systems
5. Python It's a glue language , Can be easily bonded to components in other languages in a variety of ways .