Requests crawler implements a simple web page collector

Realize a simple web page collector

• Based on the different keywords specified by Sogou, the corresponding page data is crawled

1 Introduce a new knowledge point

Parameters are dynamic ：

• If requested url Portability parameter , And if we want to dynamically operate the carried parameters, we must ：
  • 1. Encapsulate the dynamic parameters carried into a dictionary in the form of key value pairs
  • 2. Apply the dictionary to get Methodical params Parameters
  • 3. It is necessary to carry the original parameter url Delete the carried parameters

2 demonstration

Next, I will demonstrate crawling search dog search “hello Mr. tree ” Code and results

import requests

keyWord = input(" Enter a content to query ：")
#  With request parameters url, If you want to crawl the page corresponding to different keywords , We need to url Carry the parameters dynamically 
#  Realize parameter dynamic 
params = {
    
    'query': keyWord
}
url = 'https://www.sogou.com/web'
# params Parameters （ Dictionaries ）： When saving the request url Parameters carried 
response = requests.get(url=url, params=params)
page_text = response.text
fileName = keyWord + '.html'
with open(fileName, 'w', encoding='utf-8') as fp:
    fp.write(page_text)

print(fileName, ' Crawling over ！')

The screenshot of the operation shows that it has been successfully crawled down

But looking at the crawled web page, I found that it was different from the real interface

Real interface

Climb down the picture

In fact, there may be garbled code

3 solve the problem

1. First solve the problem of garbled code

   The reason for garbled code is that the written code is inconsistent with the original code . The regulations are consistent . Add... To the code

   #  Modify the coding format of the corresponding data 
   # encoding What is returned is the original encoding format of the response data , If it is assigned a value, it means that the encoding format of the response data has been modified 
   response.encoding = 'utf-8'
   

2. Solve the problem of page display 【 Exception access request 】 Cause the problem of missing request data

• Abnormal access request
  • The website background has detected that this request is not a request initiated by the browser, but a request initiated by the crawler .（ Requests that are not initiated through the browser are all exception requests ）
• How does the background of the website know whether the request is initiated through the browser ？
  • It is through the... In the request header of the decision request user-agent Judgmental
• What is? User-Agent
  • The identity of the request carrier
  • What is a request carrier ：
    • browser
      • The identity of the browser is unified and fixed , The identity can be obtained from the packet capturing tool .
    • Crawler program
      • Identity is different

Code implementation reference for solving this problem 2.6

4 How to get the browser User-Agent

1. Click in a web page F12.
2. Click on NetWork
3. Refresh web page
4. Find any requested information , Select on the right header
5. find User-Agent

5 The second anti creep mechanism

• UA testing ： The website background will detect the corresponding User-Agent, To determine whether the current request is an abnormal request .

• Anti-crawl strategy ：

  • UA camouflage : It is applied to some websites , The crawlers we write in the future will be brought by default UA Test operation .
  • Camouflage process ：
    • Capture a browser based request from the packet capturing tool User-Agent Value , Disguise it into a dictionary , Apply this dictionary to the request method （get,post） Of headers Parameters .

6 Exception access request code resolution

Take what we got from the browser User-Agent Value stored in dictionary headers in

And then headers The dictionary is assigned to the request method headers Parameters

import requests

keyWord = input(" Enter a content to query ：")
#  With request parameters url, If you want to crawl the page corresponding to different keywords , We need to url Carry the parameters dynamically 
#  Realize parameter dynamic 
params = {
    
    'query': keyWord
}
headers = {
    
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
}
url = 'https://www.sogou.com/web'
# params Parameters （ Dictionaries ）： When saving the request url Parameters carried 
#  Joined the headers, Realized UA camouflage 
response = requests.get(url=url, params=params, headers=headers)
#  Modify the coding format of the corresponding data 
# encoding What is returned is the original encoding format of the response data , If it is assigned a value, it means that the encoding format of the response data has been modified 
response.encoding = 'utf-8'
page_text = response.text
fileName = keyWord + '.html'
with open(fileName, 'w', encoding='utf-8') as fp:
    fp.write(page_text)
print(fileName, ' Crawling over ！')

Climb to success

Follow the column for more details