Twitter data collection (content, fans, keywords, etc.)

（ You can confide in me if you need me ,24 Hour online crab ）

Recently, I crawled twitter for a while because of my work .

Twitter Website is used AJAX Asynchronously loaded , use request Ask to json File data url It is also rejected

So I can only slowly simulate the browser to slide and load it slowly json It's a document （ Of course I didn't use something like selenium Class I Library , Efficiency is too low ）

for instance ：

We need to climb trump Of 1000 Twitter , Just enter the following command

GetOldTweets3 --usename "realDonaldTrump" --toptweets --maxtweets 100

The crawler results are as follows （ Because with windows System needs fanqiang, So directly in linux Under the use of ）

use python Read the code csv file （ Poor people can only buy servers without graphical interfaces ）：

import csv import pandas as pd df=pd.read_csv("/opt/test/output_got.csv") print(df)

 

 

---

 

 

Another example , We want to search on twitter “changsha” News of

GetOldTweets3 --qsearch " changsha " --maxtweets 10

No problem , If in windows You can't use it under the system. You can send me a private message or it should be blocked by a wall .

Good programmers, I think, need one click and compatibility , See what I wrote about how to use it readme That's it

Specifically github The address is as follows ：

Solin1998/SearchTT​github.com

https://github.com/Solin1998/SearchTT

 

I hope you can fork Let me have a moment , thank you ！！！！ It's important to me , Gratitude ！

---

 

Of course , Sentiment analysis , I can't do things like event extraction , Just do a part of the original data source crawler ,

It should be updated sometime to see if there is a way to bypass twitter Developer accounts, such as those who crawl for attention , Multi level attention, etc

---

I have had a private letter from an old brother several times , You can use this code to crawl through the commenters and comments ,json Format

import ssl import json import urllib.request from bs4

import BeautifulSoup from urllib.request

import urlopen

ssl._create_default_https_context = ssl._create_unverified_context

import tweepy

# Apply for this account by yourself , Sorry

consumer_key = 'lr7GQ6kTaSBkjQV'

consumer_secret = 'oxrXDT8TxsYRqIfk0k7vsX6zHyHSZ7fwZR'

access_token = '3388759955-FOU7cDJApQLDRjIbvICPCJtT5'

access_token_secret = 'qEvay5uQUxt0sTlHclSeI1KrblHJR8X'

auth = tweepy.OAuthHandler (consumer_key, consumer_secret) auth.set_access_token (access_token, access_token_secret)

api = tweepy.API (auth)

api = tweepy.API(auth, wait_on_rate_limit=True) results=api.friends(id="markturnery2k") #id That's what you're looking for user_id print(results)

Last , I also sorted out a twitter fan ranking , Such as Chinese user ranking , Hot event participation, etc ..

If you have any need, you can send a private letter to my email ：[email protected]

chat:374572744