The requests module uses

requests Module USES

requests Introduce

​ python A module based on network request of Zhongyuan , Very powerful , Simple and convenient , Very efficient , The function is to simulate the browser to send a request .

requests Module coding process

1. Appoint URL
2. Send a request
3. Obtain corresponding data
4. Persistent storage

Environmental installation

pip install requests

Actual code

demand ： Crawler Sogou homepage page data

import requests
#  One 、 Appoint url
url = 'https://www.sogou.com/'
#  Two 、 Send a request 
#  3、 ... and 、 use get The request returns a response object 
response = requests.get(url=url)
#  Get response data 
page_text = response.text
#  Four 、 Persistent storage 
with open('sougou.html', 'w', encoding='utf-8') as fp:
    fp.write(page_text)

To consolidate in action

1、 Crawl the search result page corresponding to the specified term of the search dog (get request )

import requests


# UA camouflage : The corresponding  User-Agent  Encapsulated into a dictionary （ Request header ） in 
headers = {
    
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36 Edg/98.0.1108.50'
}

url = 'https://www.sogou.com/web'
#  Handle url Parameters carried : Encapsulated in a dictionary 
kw = input(" Enter keywords \n")
param = {
    
    'query': kw
}

#  For the specified url The request initiated corresponds to url It carries parameters , And the parameters are processed in the request process 
response = requests.get(url=url, params=param, headers=headers)
page_text = response.text
fileName = kw+'.html'

#  Persistent storage 
with open(fileName, 'w', encoding='utf-8') as fp:
    fp.write(page_text)

2、 Crack Baidu translation (post request )

import json
import requests

# UA camouflage : The corresponding  User-Agent  Encapsulated in a dictionary 
headers = {
    
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36 Edg/98.0.1108.50'
}

url = 'https://fanyi.baidu.com/sug'
kw = input(' Translation input :\n')
data = {
    
    'kw': kw
}

response = requests.post(url=url, data=data, headers=headers)
#  Get response data :json Method returns a json object （ If the response data type is json Type can only be used ：Content-Type: application/json）
result = response.json()
#  Output translation results 
print(' Translation results :')
for word in result['data']:
    print(word['k'], word['v'])
#  Persistent storage 
fp = open('{}.json'.format(kw), 'w', encoding='utf-8')
json.dump(result, fp=fp, ensure_ascii=False)

3、 Climb to the ranking list of Douban films

import json
from unittest import result
import requests
url = 'https://movie.douban.com/j/chart/top_list'

# UA camouflage : The corresponding  User-Agent  Encapsulated in a dictionary 
headers = {
    
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36 Edg/98.0.1108.50'
}

param = {
    
    'type': '24',
    'interval_id': '100:90',
    'action': '',
    'start': '0',
    'limit': '100'
}

response = requests.get(url=url, params=param, headers=headers)
result = response.json()

#  Output results 
for moive in result:
    print(' ranking :{0}, The movie name :{1}'.format(moive['rank'], moive['title']))


#  Persistent storage 
fp = open('douban.json', 'w', encoding='utf-8')
json.dump(result, fp=fp, ensure_ascii=False)

json Formatting tool