Scripy learning

scrapy introduction

1.scrapy brief introduction

Scrapy It's based on Twisted The asynchronous processing framework of , Is pure python Implementation of the crawler framework . You can quickly grab data with a small amount of code .
Scrapy Is applicable to Python A quick 、 High level screen grabs and web Grabbing framework , Used to grab web Site and extract structured data from the page .Scrapy A wide range of uses , Can be used for data mining 、 Monitoring and automated testing . Anyone can modify it conveniently according to their needs . It also provides a base class for many types of reptiles , Such as BaseSpider、sitemap Reptiles, etc , The latest version offers web2.0 Reptile support .

2.Scrapy Framework and function

2.1 Frame diagram

2.2 Function of each part

name	function
Scrapy Engine（Scrapy engine ）	Scrapy The engine is the core of the framework , be responsible for Spider、ItemPipeline、Downloader、Scheduler Intermediate communication , The signal 、 Data transfer, etc
Spiders( Reptiles )	Be responsible for handling all messages sent by the engine Response, Extract data from , extract URl, And submit to the engine
Scheduler( Scheduler )	Responsible for receiving the engine sent Request request
Downloader( Downloader )	Responsible for downloading Scrapy Engine( engine ) All sent Requests request , And get it Responses Return to Scrapy Engine( engine ), Engine to Spider To deal with it .
Item Pipeline( Project pipeline )	Be responsible for the data sent by the engine , And do post-processing （ Data analysis , Data storage, etc ）

3.Scrapy install

3.1 Installation command

windows Next ：

	pip install Scrapy

Check whether the installation is successful ：

scrapy startProject Project name

You can start your first spider with:
First step ： cd myspider
The second step ：scrapy genspider example（ Reptile name ） example.com（ The website you want to crawl ）

For website xxxx Instead of

After writing the code ： Execute the crawler

scrapy crawl Reptile name