Request attribute in crawler

Review reptiles ····
Learning source is wind change programming （ Strong wind driven transformer ）
And the lovely Songtian teacher of Beijing University of Technology ！

Basic introduction

3steps
Access to web pages ——— Parse web pages —— Store the data

Initiate network request , Get web information ———— from HTML Extract the desired data from the source code ———— Store the parsed data

use requsets The library initiates a network request

import requests

res =  requests.get(' website ')
#  call requests.get() function , Fill in the parameter position URL, Assign the result to a variable res On 

# see res The content of 
print（res）
# Use  type（） Function to see  requests.get()  The type of the result returned by the function 
res_type = type(res)
print(res_type)
# requests.get()  The function returns a result that belongs to  requests.models.Response  Class object 

#  Print Response Object's status_code attribute , Status code 
print(res.status_code)

Response.text attribute

text = res.text

Response.text You can get the string form of web page code
use Response.text Attribute can get specific content

Response.encoding attribute

Previous variables res Is an object of a class
Objects have many properties

res.encoding = 'utf-8'

http And https The difference between

http Agreement , The data is not encrypted , So the account number and password entered on the network , Personal information and other data are very dangerous if they are obtained .
and https The agreement will be in http Encrypt the data based on the Protocol , be relative to http It's safer for me 、 reliable .

see robots agreement

Add /robots.txt that will do

User-agent:*
Disallow:/
# * Number means everything , No one is allowed to climb here