The use of scrapy crawler framework

First let's download the scrapy package

Select Terminal to create a scrapy project (scrapy startproject project name)

Now that the project is created, we need to modify the settings.py file

Line 20, ROBOTSTXT_OBEY = False

Uncomment 65 to 67 (it was originally commented)

Uncomment 40 to 43 (it was originally commented), and add the request header inside,

Enter the spiders directory to create a crawler file

scrapy genspider crawler file name domain name

We crawl the course name and number of learners

Modify start_urls to the address to be crawled

Select all copy and paste

Find the parent tag to crawl all information, and the parent tag is unique (you can right-click, view the source code of the web page, Ctrl+f search whether there is only one attribute of the tag)

Through analysis, the information we want to crawl is in the li of ul

We use Xpath to get all li tags

Traverse il to locate the content to be crawled