Usage of requests module

python The libraries that initiate requests for web pages in urllib and requests,Python Reptilian time , More recommended requests library . because requests Than urllib More convenient . that request The function of the library is to simulate the browser to send a request to the web page .

How to use it ？ Can be divided into 5 Step .

1、 Appoint url

2、 Send a request

3、 Get response data

4、 Parsing data

5、 Save the data

Programming begins

First, in the pycharm Download requests library , In the file -> Set up -> project find python Interpreter

Click on the top left + Number , Input requests, Then install .

Build a new project , Create a new one python file , Enter... On the first line # -- coding:UTF-8 --

This is a python Document coding statement , To illustrate your Python The source program file is encoded with the code used . By default, your program needs to use ascii Code to write , But if you write Chinese in it ,python The interpreter usually reports an error , But if you add the file code you use ,python It will be processed automatically and no error will be reported .

Simply put, you can write Chinese without reporting errors . Then import requests library .

import requests

  After importing the library, you can divide it according to the above 5 Step start programming , Here is a simple way to get Sogou Page source code to introduce .

# -- coding:UTF-8 --
import requests

if __name__ == "__main__":
    #1、 designated url
    url = 'https://www.sogou.com/'
    #2、 Send a request 
    #get Method will return a response object 
    response = requests.get(url=url)
    #3、 Get response data .text What is returned is the response data in the form of string 
    html = response.text
    #4、 Data analysis 
    print(html)
    #5、 Save the data 
    with open('./sogou.html','w',encoding="utf-8") as fp:
        fp.write(html)
    print(' End of crawling data ')

The second step can be get Mode request , It can also be post Mode request .

The third step requests After sending the request , The server will return a response , and requests Usually, the response content is automatically decoded .

response.text Get the response content in the form of text

requests.content Get the response content in binary form

requests.json() Is to obtain json The way of form

requests.raw You can get the original data stream , use .read() You can read the contents

The fourth step is to simply print the data , If you want some data in the web page, you can use xpath、beautifulsoup、 Regular expressions are filtered out .

The fifth step is to simply save Sogou Source data .

requests Library has many other uses , You can have a look at requests Official document of the library

Document address ：https://docs.python-requests.org/zh_CN/latest/