当前位置:网站首页>Implementation of crawling web pages and saving them to MySQL using the scrapy framework
Implementation of crawling web pages and saving them to MySQL using the scrapy framework
2022-07-07 15:17:00 【1024 questions】
Hello everyone , In this issue, abin will share with you Scrapy Crawler framework and local Mysql Use . Today, the web page ah bin crawled is Hupu Sports Network .
(1) Open Hupu Sports Network , Analyze the data of the web page , Use xpath Positioning elements .
(2) After the first analysis of the web page, start creating a scrapy Reptile Engineering , Execute the following command at the terminal :
“scrapy startproject huty( notes :‘hpty’ Is the name of the crawler project )”, Get the project package shown in the figure below :
(3) Enter into “hpty/hpty/spiders” Create a crawler file called ‘“sww”, Execute the following command at the terminal : “scrapy genspider sww” (4) After the first two steps , Edit the crawler files related to the whole crawler project . 1、setting Editor of the document :
The gentleman agreement was originally True Change it to False.
Then open this line of code that was originally commented out .
2、 Yes item File for editing , This file is used to define data types , The code is as follows :
# Define here the models for your scraped items## See documentation in:# https://docs.scrapy.org/en/latest/topics/items.htmlimport scrapyclass HptyItem(scrapy.Item): # define the fields for your item here like: # name = scrapy.Field() players = scrapy.Field() The team = scrapy.Field() ranking = scrapy.Field() Average score = scrapy.Field() shooting = scrapy.Field() Three-point percentage = scrapy.Field() Free throw percentage = scrapy.Field()
3、 Edit the most important crawler files ( namely “hpty” file ), The code is as follows :
import scrapyfrom ..items import HptyItemclass SwwSpider(scrapy.Spider): name = 'sww' allowed_domains = ['https://nba.hupu.com/stats/players'] start_urls = ['https://nba.hupu.com/stats/players'] def parse(self, response): whh = response.xpath('//tbody/tr[not(@class)]') for i in whh: ranking = i.xpath( './td[1]/text()').extract()# ranking players = i.xpath( './td[2]/a/text()').extract() # players The team = i.xpath( './td[3]/a/text()').extract() # The team Average score = i.xpath( './td[4]/text()').extract() # score shooting = i.xpath( './td[6]/text()').extract() # shooting Three-point percentage = i.xpath( './td[8]/text()').extract() # Three-point percentage Free throw percentage = i.xpath( './td[10]/text()').extract() # Free throw percentage data = HptyItem( players = players , The team = The team , ranking = ranking , Average score = Average score , shooting = shooting , Three-point percentage = Three-point percentage , Free throw percentage = Free throw percentage ) yield data
4、 Yes pipelines File for editing , The code is as follows :
# Define your item pipelines here## Don't forget to add your pipeline to the ITEM_PIPELINES setting# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html# useful for handling different item types with a single interfacefrom cursor import cursorfrom itemadapter import ItemAdapterimport pymysqlclass HptyPipeline: def process_item(self, item, spider): db = pymysql.connect(host="Localhost", user="root", passwd="root", db="sww", charset="utf8") cursor = db.cursor() players = item[" players "][0] The team = item[" The team "][0] ranking = item[" ranking "][0] Average score = item[" Average score "][0] shooting = item[" shooting "] Three-point percentage = item[" Three-point percentage "][0] Free throw percentage = item[" Free throw percentage "][0] # Three-point percentage = item[" Three-point percentage "][0].strip('%') # Free throw percentage = item[" Free throw percentage "][0].strip('%') cursor.execute( 'INSERT INTO nba( players , The team , ranking , Average score , shooting , Three-point percentage , Free throw percentage ) VALUES (%s,%s,%s,%s,%s,%s,%s)', ( players , The team , ranking , Average score , shooting , Three-point percentage , Free throw percentage ) ) # Commit transaction operations db.commit() # Close cursor cursor.close() db.close() return item
(5) stay scrapy After the frame is designed , Come first mysql Create a file called “sww” The database of , Create a database named “nba” Data sheet for , The code is as follows : 1、 Create database
create database sww;
2、 Create data table
create table nba ( players char(20), The team char(10), ranking char(10), Average score char(25), shooting char(20), Three-point percentage char(20), Free throw percentage char(20));
3、 You can see the structure of the table by creating the database and data table :
(6) stay mysql After creating the data table , Return to the terminal again , Enter the following command :“scrapy crawl sww”, The result
This is about using Scrapy The frame crawls the web page and saves it to Mysql This is the end of the article on the implementation of , More about Scrapy Crawl the web page and save the content. Please search the previous articles of SDN or continue to browse the relevant articles below. I hope you will support SDN in the future !
边栏推荐
猜你喜欢
#HPDC智能基座人才发展峰会随笔
Window环境下配置Mongodb数据库
IDA pro逆向工具寻找socket server的IP和port
CTFshow,信息搜集:web2
简述keepalived工作原理
CTFshow,信息搜集:web12
With 8 modules and 40 thinking models, you can break the shackles of thinking and meet the thinking needs of different stages and scenes of your work. Collect it quickly and learn it slowly
CTFshow,信息搜集:web7
居然从408改考自命题!211华北电力大学(北京)
写一篇万字长文《CAS自旋锁》送杰伦的新专辑登顶热榜
随机推荐
最安全的证券交易app都有哪些
Niuke real problem programming - Day11
【数据挖掘】视觉模式挖掘:Hog特征+余弦相似度/k-means聚类
CTFshow,信息搜集:web3
使用cpolar建立一个商业网站(2)
HW初级流量监控,到底该怎么做
Jetson AGX Orin CANFD 使用
Bits and Information & integer notes
Stm32f103c8t6 PWM drive steering gear (sg90)
Ctfshow, information collection: web8
拜拜了,大厂!今天我就要去厂里
2022年5月互联网医疗领域月度观察
Niuke real problem programming - day13
CTFshow,信息搜集:web12
[data mining] visual pattern mining: hog feature + cosine similarity /k-means clustering
CTFshow,信息搜集:web1
[make a boat diary] [shapr3d STL format to gcode]
Read PG in data warehouse in one article_ stat
Ctfshow, information collection: Web3
How bad can a programmer be? Nima, they are all talents