Reptile exercises (III)

subject : Webpage "https://www.6pian.cn/xq.html"

 1. Need to crawl the first six pages of the page source code

 2. preservation

'''
1. Analyze the web :
https://www.6pian.cn/
https://www.6pian.cn/xq.html
https://www.6pian.cn/xq/1/0.html
https://www.6pian.cn/xq/2/0.html
https://www.6pian.cn/xq/3/0.html
'''


import urllib.request

start = int(input(" Please enter the start page "))
end = int(input(" Please enter the end page "))
headers = {
 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36 Edg/103.0.1264.44'
}
for n in range(start, end + 1):
 url = 'https://www.6pian.cn/xq/{}/0.html'.format(n)
 print(url)
 q = urllib.request.Request(url,headers=headers)
 response = urllib.request.urlopen(q)
 with open(f' The first {n} page .html','w',encoding='utf-8')as f:
 f.write(response.read().decode('utf-8'))

First , Analyze the links of niupian webpage , Find the rules of web pages , structure URL

The page links of each page are different , Be sure to check carefully , Find the rules , Use them separately input Import start and end pages , And receive... With variables , use headers camouflage , There are several points in the page to check , Find the package of the whole web page , Click open and find user agent, Copied to the headers Inside , Sometimes you need to copy cookie, But there is no need to use

Input the cycle to ensure that one side climbs , Then climb the next page , So your formula must be written correctly , Only with clear logic can we write the correct answer , The answer cannot die , Carry a hard cover , The next step is to build URL, Put variables n Pass in , Replace variables in the link , Make the whole link , Meet all page links , Once the crawl fails , You have to re-examine your own construction , Is there a problem ？ Can it meet the links of other pages , It's better to print it later , The website you just built , After ensuring that there is no error, you can instantiate the request object , It's here to see Request How many variables are there in it, and one variable is passed in to receive the whole instantiation process

With open Used to save data in the second quotation mark ,w For writing. If not, the default is r As read-only , When transcoding, you must pay attention to the wrong code type, which will definitely report an error , So you can be in pycharm Get the web source code of six pages in .

User-Agent ad locum ：

Data package of the whole page ：