Crawler: from entry to imprisonment (II) -- Web collector

Go straight to the theme

        at present python3 There are already very perfect requests modular , We mainly use this module to crawl web data

requests modular ：python A module based on network request of Zhongyuan , Very powerful , Simple and convenient , Very efficient .
effect ： Simulate browser to send request .

How to use ：（requests Module coding process ）
    - Appoint url
    - Initiate request
    - Get response data
    - Persistent storage

First , Open the terminal installation environment

pip install requests

After the environment is installed , We can formally start to build the body of crawl code

Create a python file , Import requests modular  

import requests
if __name__ == "__main__":

The target site's URL The address is stored in a string variable

Must be based on “ character string ” stored

URL=" The target URL you crawled URL"

utilize requests Modular get Method to send a request to the target web page

result=requests.get(url=URL)

Then store the content of the web page in the file of the local corresponding format

with open("./test.html","w",encoding="utf-8") as fp
    fp.write(result.text)

Print end prompt

print(" Crawling over ！")

  A simple web page raw data acquisition device is done , You can preview the crawled web page locally

Add a concept ！

Ajax asked ：Ajax namely Asynchronous Javascript And XML（ asynchronous JavaScript and XML） stay 2005 By the Jesse James Garrett New terminology proposed , Used to describe a technology that uses a collection of existing technologies ‘ new ’ Method , Include : HTML  or  XHTML, CSS, JavaScript, DOM, XML, XSLT, And most importantly XMLHttpRequest. [3]   Use Ajax Technology web applications can quickly present incremental updates to The user interface On , Instead of overloading （ Refresh ） The entire page , This allows the program to respond more quickly to user actions .