当前位置:网站首页>初学爬虫-笔趣阁爬虫
初学爬虫-笔趣阁爬虫
2022-07-02 04:35:00 【weixin_43446292】
import requests
from lxml import etree
base_url=input(“请输入小说url:”) #如春日宴的url为https://www.xbiquge.la/20/20671/
headers={
“User-Agent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:101.0) Gecko/20100101 Firefox/101.0”
}
response=requests.get(base_url,headers=headers)
r=response.content.decode()
r1=etree.HTML #将字符串格式的 html 片段解析成 html 文档
r2=r1.xpath(‘//[@id=“info”]/h1/text()‘) #获取小说名字
r3=r1.xpath(’//[@id=“list”]/dl/dd/a/text()’) #获取小说每章标题
r4=r1.xpath(‘//[@id=“list”]/dl/dd/a/@href’) #获取每章节链接
r5=r1.xpath('//[@id=“info”]/p[1]/text()’) #获取作者名字
r6=r1.xpath(‘//[@id=“intro”]/p[2]/text()‘) #获取文案
r7=’‘.join(r5).split(’:‘)[1] #获取作者名字
chapter_list=[]
for i in r4:
url=“https://www.xbiquge.la”+i
chapter_list.append(url) #合成每章节的url
for i in r2:
title=’{}by{}.txt’.format(i,r7) #获取保存时txt的名字
content_list=[]
with open(title,“a”,encoding=“utf-8”) as f:
f.writelines(r6)
f.write(‘\n’)
for (x,y) in zip(chapter_list,r3):
response2=requests.get(x)
res=response2.content.decode()
res1=etree.HTML(res)
res3=res1.xpath('//[@id=“content”]/text()’) #获取每章节小说内容
f.writelines(y)#写入每章节的标题
f.write(‘\n’)
f.writelines(res3)#写入每章节的小说内容
f.write(‘\n’)
print(“{}采集完毕,共{}章节”.format(title,len(chapter_list)))
边栏推荐
- Landing guide for "prohibit using select * as query field list"
- [source code analysis] NVIDIA hugectr, GPU version parameter server - (1)
- LeetCode-对链表进行插入排序
- Pytorch-Yolov5從0運行Bug解决:
- Learn AI safety monitoring project from zero [attach detailed code]
- Mysql database learning
- [understand one article] FD_ Use of set
- cookie、session、tooken
- 二叉树解题(一)
- Li Kou interview question 02.08 Loop detection
猜你喜欢
Use a mask to restrict the input of the qlineedit control
Play with concurrency: draw a thread state transition diagram
Typescript practice for SAP ui5
Thinkphp Kernel wo system source Commercial Open source multi - user + multi - Customer Service + SMS + email notification
Several methods of capturing packets under CS framework
Mysql database learning
Www2022 | know your way back: self training method of graph neural network under distribution and migration
Research on the security of ognl and El expressions and memory horse
[JS event -- event flow]
The core idea of performance optimization, dry goods sharing
随机推荐
LCM of Spreadtrum platform rotates 180 °
社交媒体搜索引擎优化及其重要性
Federal learning: dividing non IID samples according to Dirichlet distribution
The confusion I encountered when learning stm32
Common locks in MySQL
Okcc why is cloud call center better than traditional call center?
Ten thousand volumes are known to all, and one page of a book is always relevant. TVP reading club will take you through the reading puzzle!
Websites that it people often visit
【毕业季·进击的技术er】年少有梦,何惧彷徨
Pytorch-Yolov5從0運行Bug解决:
cs架构下抓包的几种方法
Record the bug of unity 2020.3.31f1 once
powershell_ View PowerShell function source code (environment variable / alias) / take function as parameter
What methods should service define?
Pytoch --- use pytoch for image positioning
Pytoch yolov5 runs bug solution from 0:
My first experience of shadowless cloud computer
CorelDRAW Graphics Suite2022免费图形设计软件
office_ Delete the last page of word (the seemingly blank page)
Yolov5 network modification tutorial (modify the backbone to efficientnet, mobilenet3, regnet, etc.)