Scrapy framework is introduced

Table of Contents

1. Introduction

2. Why use scrapy

3.scrapy features

4. Advantages

5. Flowchart

---

1.Introduction

1) scrapy is a fast, high-level screen scraping and web scraping framework developed in python for scraping web sites and extracting structured data from pages.scrapy=scrach+python

2) scrapy is widely used and can be used as an application framework for extracting structured data in a large number of applications such as data mining, monitoring and automated testing, information processing and historical archives, and is widely used in enterprises

3) scrapy uses the asynchronous network library twisted to handle network communication, with a clear structure and various middleware interfaces, which can flexibly complete variouskind of demand.scrapy is a popular python event-driven networking framework written by twisted that uses non-blocking asynchronous processing

2.Why use scrapy

1. Easier to build and large scale scraping projects

2. The built-in mechanism is called a selector for extracting data from a website (web page)

3. Asynchronous processing of requests, very fast

4. The crawling speed can be automatically adjusted using the automatic adjustment mechanism

5. Ensure developer accessibility

3.scrapy features

1. Is an open source, free to use web crawler framework

2.scrapy generated format export such as: JSON, CSV, XML

3. Built-in from source code, use xpath or css selector to extract data

4.scrapy is crawler-based and allows to extract data from web pages in an automated way

4.Benefits

1. It is easy to expand, fast and powerful

2. This is a cross-platform application framework

3.scrapy request scheduling and asynchronous processing

4.scrapy comes with a built-in service called scrapyd that allows using JSON webspan>Project and control spiders on the service

5. Ability to scrape any website, even if the website does not have element data access api

5. Flowchart

Handwritten:

 

scrapy engine (engine): Otherwise, the communication between spider, itempipeline, Downloader, scheduler, signal, data transmission, etc.

scheduler (scheduler): responsible for accepting requests sent by engine, and sorting and sorting in a certain way, entering the queue, when Engine hand over to engine

when needed

Downloader: Responsible for downloading all requests sent by engine, and returning the obtained response to engine, which is handed over to spider by the engine for processing

spider (crawler): responsible for processing all responses, analyzing and extracting data from it, obtaining the data required by the item field, and submitting the url that needs to be followed up to the engine , enter Scheduler

again

item pipeline (pipeline): responsible for processing the items obtained in spider and performing post-processing (detailed analysis, filtering, storage, etc.)place

Downloader middlewares (download middleware): can be used as a component that can customize and extend the download function

spider middlewares (spider middleware): can be understood as a custom extension and operation engine and spider intermediate communication functional components (such as entering the response of spider and responding from spider outgoing request)

The simplest single web crawling process is: spiders ->scheduler -> downloader -> spiders -> item pipeline

Attention!Only when there is no request in the scheduler, the whole program will stop, that is to say, for the url that fails to download, scrapy will also download it again.