UA camouflage, get and post in requests carry parameters to obtain JSON format content

First of all, let's learn an anti climbing strategy UA camouflage .

UA yes User-Agent（ The identity of the request carrier ）

Most websites have one UA Anti creep mechanism of detection , It will detect the identity of the request carrier , If it is detected that the identity of the request carrier is a browser , It indicates that the request is normal , Conversely, if the detection of identity is not browser based , That's reptile , It is likely that the server will reject the request .

UA camouflage ： Let the crawler's identity disguise as a browser .

How to disguise ？

First open the browser , Right click to check or press fn and f12 Open the developer tool

Then choose the network , Select a request header , Turn down and find it User-Agent Then copy and package it into a dictionary .

headers = {
    "user-agent": "Mozilla / 5.0(Windows NT 10.0;Win64;x64) AppleWebKit / 537.36(KHTML, likeGecko) Chrome / 97.0.4692.71Safari / 537.36Edg / 97.0.1072.55"
}

Put it in when you make a request

page_text = requests.get(url=url,headers=headers)

Next, deal with if url How to carry parameters in .

It also encapsulates parameters into a dictionary , however get The request is assigned to params,post The request is assigned to data.

response = requests.get(url=url,params=param,headers=headers)

response = requests.post(url=url,data=data,headers=headers)

  Last , How to get json Formatted data

json() The object returned （obj）, Only confirm that the response content is json Type of , Can be used json() Method .

How do you know if the response content is json Type? ？ Or open the developer tool , Select the network , Select the request header to find content-type, You can know which type it is .

The following is the code of crawling Baidu translation

# -- coding:UTF-8 --
import json

import requests

if __name__ == "__main__":
    url = 'https://fanyi.baidu.com/sug'
    word = input('enter a word:')
    data = {
        'kw':word
    }
    headers = {
        "user-agent": "Mozilla / 5.0(Windows NT 10.0;Win64;x64) AppleWebKit / 537.36(KHTML, likeGecko) Chrome / 97.0.4692.71Safari / 537.36Edg / 97.0.1072.55"
    }

    response = requests.post(url=url,data=data,headers=headers)
    dic_obj = response.json()
    filename = word+'.json'
    with open (filename,'w',encoding='utf-8') as fp:
        json.dump(dic_obj,fp=fp,ensure_ascii=False)
    print('over')

Use json.dump Import when json library ,dump() The role of the python The object is encoded as Json character string .

ensure_ascii： This parameter takes only Boolean values . If it is not set to true, Default output ASCLL value , If you put ensure_ascii Assigned as False, You can output Chinese .