Twitter Scraper
Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse-engineered. No API rate limits. No tokens needed. No restrictions. Extremely fast.
You can use this library to get the text of any user's Tweets trivially.
Installation
go get -u github.com/n0madic/twitter-scraper
Usage
Get user tweets
package main
import (
"context"
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
for tweet := range scraper.GetTweets(context.Background(), "Twitter", 50) {
if tweet.Error != nil {
panic(tweet.Error)
}
fmt.Println(tweet.Text)
}
}
It appears you can ask for up to 50 tweets (limit ~3200 tweets).
Search tweets by query standard operators
Tweets containing “twitter” and “scraper” and “data“, filtering out retweets:
package main
import (
"context"
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
for tweet := range scraper.SearchTweets(context.Background(),
"twitter scraper data -filter:retweets", 50) {
if tweet.Error != nil {
panic(tweet.Error)
}
fmt.Println(tweet.Text)
}
}
The search ends if we have 50 tweets.
See Rules and filtering for build standard queries.
Set search mode
scraper.SetSearchMode(twitterscraper.SearchLatest)
Options:
twitterscraper.SearchTop- default modetwitterscraper.SearchLatest- live modetwitterscraper.SearchPhotos- image modetwitterscraper.SearchVideos- video mode
Get profile
package main
import (
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
profile, err := scraper.GetProfile("Twitter")
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", profile)
}
Get trends
package main
import (
"fmt"
twitterscraper "github.com/n0madic/twitter-scraper"
)
func main() {
scraper := twitterscraper.New()
trends, err := scraper.GetTrends()
if err != nil {
panic(err)
}
fmt.Println(trends)
}
Use http proxy
err := scraper.SetProxy("http://localhost:3128")
if err != nil {
panic(err)
}
Load timeline with tweet replies
scraper.WithReplies(true)
Default Scraper (Ad hoc)
In simple cases, you can use the default scraper without creating an object instance
import twitterscraper "github.com/n0madic/twitter-scraper"
// for tweets
twitterscraper.GetTweets(context.Background(), "Twitter", 50)
// for tweets with replies
twitterscraper.WithReplies(true).GetTweets(context.Background(), "Twitter", 50)
// for search
twitterscraper.SearchTweets(context.Background(), "twitter", 50)
// for profile
twitterscraper.GetProfile("Twitter")
// for trends
twitterscraper.GetTrends()
