This is probably the most comprehensive project about Twitter information crawler search on the Chinese Internet

Recently, I crawled twitter for a while because of my work .

Twitter Website is used AJAX Asynchronously loaded , use request Ask to json File data url It is also rejected

So I can only slowly simulate the browser to slide and load it slowly json It's a document （ Of course I didn't use something like selenium Class I Library , Efficiency is too low ）

for instance ：

We need to climb trump Of 1000 Twitter , Just enter the following command

GetOldTweets3 --username "realDonaldTrump" --toptweets --maxtweets 100

The crawler results are as follows （ Because with windows System needs fanqiang, So directly in linux Under the use of ）

use python Read the code csv file （ Poor people can only buy servers without graphical interfaces ）：

 

import csv
import pandas as pd
df=pd.read_csv("/opt/test/output_got.csv")
print(df)

 

 

---

 

 

Another example , We want to search on twitter “changsha” News of

GetOldTweets3 --querysearch " changsha " --maxtweets 10

No problem , If in windows You can't use it under the system. You can send me a private message or it should be blocked by a wall .

Good programmers, I think, need one click and compatibility , See what I wrote about how to use it readme That's it

 

Specifically github The address is as follows ：

Solin1998/SearchTT​github.com

 

I hope you can fork Let me have a moment , thank you ！！！！

---

 

Of course , Sentiment analysis , I can't do things like event extraction , Just do a part of the original data source crawler ,

It should be updated sometime to see if there is a way to bypass twitter Developer accounts, such as those who crawl for attention , Multi level attention, etc