Reptile exercises (II)

"""
The target site ：https://www.1ppt.com/moban/
Climbing requirements ：
1、  Turn the page and crawl the source code on this page
2、  And save it locally , Pay attention to coding
"""
'''
1. Analysis website ：
https://www.1ppt.com/moban/  first page
https://www.1ppt.com/moban/ppt_moban_2.html  The second page
https://www.1ppt.com/moban/ppt_moban_3.html  The third page
'''

import urllib.request

start = int(input(" Enter the start page ")) #  turn int
end = int(input(" Enter the end page "))
headers = {
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'
}
for n in range(start, end + 1):
 url = 'https://www.1ppt.com/moban/ppt_moban_{}.html'.format(n)
 print(url)
 req = urllib.request.Request(url, headers=headers) #  Instantiate the request object
 response = urllib.request.urlopen(req) #  The method to send the request
 with open(f' The first {n} page .html', 'a', encoding='gb2312') as f:
 f.write(response.read().decode('gb2312'))

First , Let's open the website , Right click to view the source code of the web page , Or right click to check the packet capture ,Network The web is where web packets are displayed , Find a packet with the same URL , It opens at response Response content , This is also the page source code .

We are pycharm New file , The first step is to analyze the Web Capture , Manual page turning , find url law , Take down the web link, copy and paste it in pycharm In engineering documents , We can see that the links of the first page and the following pages are inconsistent , You can manually add , Judgment may be needed later .

then , We use it import Import urllib.request modular ,Ctrl+Alt+L It can make the code standardized .

structure url, You can use circulation , You can also build page by page , The URL was http(80)https(443) agreement , Ports in brackets , Then comes the domain name ,“？” After is get Some parameters carried by the request ,kw Search content ,pn For page , Other parameters may not affect the crawl process much , You can delete . Build up URL after , You can print

Initiate request

zlf = urllib.request.Request(url,headers=headers), Build the request object , Instantiate the request object , And receive... With variables

response = urllib.request.urlopen(zlf)

Get the response content , Last , use with open preservation , file name ,'w' write in ( The mode is read-only by default 'r'),encoding( Coding format , How to decode how to write ),as Take an alias later ,

 read Extract data in byte form , You can transcode later decode