1. Get data - requests.get()

1. The working principle of crawler

Get data – parse data – extract data – store data

2. Get data

The essence is to send a request to the server through the URL, and the server encapsulates the relevant content into a Response object and returns it to us, which is achieved through requests.get().There are four commonly used methods (status_code, content, text, encoding) under the Response object we obtained.

3. requests.get()

import requests #Introduce requests module

res = requests.get('url') # Request data from the server, the result returned by the server is a Response object

print(type(res)) # Terminal display:

This means res is an object of the requests.models.Response class.

3. response.status_code

Usage: print(variable.status_code),

It is used to check whether the request is responded correctly. If the response status code is 200, it means the request is successful.

The response status code indicates the server's response to the request.For example, 200 means the server responded successfully, 403 means access is forbidden, 404 means the page was not found, and 408 means the request timed out.The browser will make corresponding processing according to the status code.In the crawler, the status of the server can be judged according to the status code. If the status code is 200, continue to process the data, otherwise ignore it directly.

4. response.content

response.content can return the content of the Response object in the form of binary data, which is suitable for downloading pictures, audio and video. Example:

import requests

#Image address
URL=‘‘https://img1.baidu.com/it/u=2076064484,1314795796&fm=253&fmt=auto&app=120&f=JPEGw=531&h=309’’

Send the request and put the returned result in the variable res

res = requests.get(url)
# Return the content of the Reponse object as binary data
pic = res.content

# Download a picture file and name it picture.jpg. The content of the picture needs to be written only in binary wb.
with open(r'C:UsersAveryDesktop estpicture.jpg', 'wb') as f:

Get the binary content of pic and write f

f.write(pic)

In this way, our pictures are downloaded successfully!

5. response.text

response.text This method can return the content of the Response object in the form of a string, which is suitable for downloading text and webpage source code.Here's an example:

import requests

Article address

url = 'https://localprod.pandateacher.com/python-manuscript/crawler-html/sanguo.md'

Send the request and put the returned result in the variable res

res = requests.get(url)
# Return the content of the Response object as a string
novel = res.text
#Print variables
print(novel[0:170])

6. response.encoding

response.encoding method, it can help us define the encoding of the Response object, the example is as follows:

import requests

Article address

url = 'https://localprod.pandateacher.com/python-manuscript/crawler-html/sanguo.md'

Send the request and put the returned result in the variable res

res = requests.get(url)
# Define the encoding corresponding to the response as utf-8
res.encoding = 'utf-8'

Returns the content of the Response object as a string

novel = res.text

Print variables

print(novel[0:170])