Crawler crawls Sina Weibo data

Tools ： Cloud gathering reptile

The goal is ： Capture all microblogs of a blogger

Analyze the structure of the web page ：

The idea of our crawling is to simulate the browser to automatically access the page crawling .

Let's take a look at the page structure , First, each Weibo list , Three or four pull-down loads are required , If there is a page turning button at the bottom , Then judge that this page is loaded .

 

 

Login problem

To crawl, you need to log in , How to login ？

No verification code is required for login , If you make a mistake , Will ask you to enter the verification code , So there is no technical difficulty in logging in .

We can create one 【 Login module 】, First log in with a browser , In the future, all pages will be shared based on this browser cookie Go grab it .

 

 

Flow chart design ：

 

 

 

We don't need the details page of Weibo . So there is no detail page for the whole crawler process , The data is extracted from the list .

Crawling results ：

Total cost 5 Minutes of time , Grab it 10 A page , 400 microblogs in total . Because my microblog is not posted very often .

The data are as follows ：

 

 

Make a simple word cloud ：