Python snscrape: How to scrape tweet URL/link using snscrape?


What parameter should we use to include the URL/link of tweets? I have here the date, username, and content. Another question also is how can we transform the date in the dataframe into GMT+8? The timezone is in UTC. Please see code below for reference:

import snscrape.modules.twitter as sntwitter
import pandas as pd

query = "(from:elonmusk) until:2023-01-28 since:2023-01-27"
tweets = []
limit = 100000

for tweet in sntwitter.TwitterSearchScraper(query).get_items():
    if len(tweets) == limit:
        tweets.append([, tweet.username, tweet.content])
df = pd.DataFrame(tweets, columns=['Date', 'Username', 'Tweet'])

#Save to csv
Asked By: jislesplr



The get_items() return every single a search result with class type.
So the count of tweets needs to count by for loop.

This code will works

100K tweets is possible but it take too much time, I reduced 1K tweets.

import snscrape.modules.twitter as sntwitter
import pandas as pd

query = 'from:elonmusk since:2022-08-01 until:2023-01-28'
limit = 1000

tweets = sntwitter.TwitterSearchScraper(query).get_items()

index = 0
df = pd.DataFrame(columns=['Date','URL' ,'Tweet'])

for tweet in tweets:
    if index == limit:
    URL = "{0}/status/{1}".format(tweet.user.username,
    df2 = {'Date':, 'URL': URL, 'Tweet': tweet.rawContent}
    df = pd.concat([df, pd.DataFrame.from_records([df2])])
    index = index + 1

# # Converting time zone from UTC to GMT+8
df['Date'] = df['Date'].dt.tz_convert('Etc/GMT+8')


This single data of get_items()
it needs to extract only required key’s value -> Date> URL

tweet.rawContent-> Tweet
  "_type": "snscrape.modules.twitter.Tweet",
  "url": "",
  "date": "2023-01-28T02:44:31+00:00",
  "rawContent": "@tn_daki @ShitpostGate Yup",
  "renderedContent": "@tn_daki @ShitpostGate Yup",
  "id": 1619164489710178307,
  "user": {
    "_type": "snscrape.modules.twitter.User",
    "username": "elonmusk",
    "id": 44196397,
    "displayname": "Mr. Tweet",
    "rawDescription": "",
    "renderedDescription": "",
    "descriptionLinks": null,
    "verified": true,
    "created": "2009-06-02T20:12:29+00:00",
    "followersCount": 127536699,
    "friendsCount": 176,
    "statusesCount": 22411,
    "favouritesCount": 17500,
    "listedCount": 113687,
    "mediaCount": 1367,
    "location": "",
    "protected": false,
    "link": null,
    "profileImageUrl": "",
    "profileBannerUrl": "",
    "label": null,
    "url": ""
    ... cut off


                        Date                                                URL                                              Tweet
0  2023-01-27 15:29:36-08:00                                  @farzyness No way
0  2023-01-27 15:14:05-08:00  @mtaibbi Please correct your bs @PolitiFact &a...
0  2023-01-27 14:52:55-08:00  @WallStreetSilv A quarter of all taxes just to...
0  2023-01-27 13:28:26-08:00           @nudubabba @mikeduncan Yeah, whole thing
0  2023-01-27 13:12:16-08:00  @TaraBull808 That’s way more monkeys than the ...
..                       ...                                                ...                                                ...
0  2022-12-14 11:14:53-08:00  @Jason Advertising revenue next year will be l...
0  2022-12-14 04:08:43-08:00                        @Balyx_ He would be welcome
0  2022-12-14 03:42:47-08:00  @NorwayMFA @TwitterSupport @jonasgahrstore @AH...
0  2022-12-14 03:35:14-08:00                                    @AvidHalaby Wow
0  2022-12-14 03:35:03-08:00                     @AvidHalaby Live & learn …

[1000 rows x 3 columns]


Converting time zone pandas dataframe

Tweet URL format

Detain information in here


URL = ""

It saved into csv file.

0,2023-01-27 15:14:05-08:00,,@mtaibbi Please correct your bs @PolitiFact & @snopes

It matched the tweet content and pandas Tweet column.

Also, you can add column, followers Count, friends Count, statuses Count, favourites Count, listed Count, media Count, reply Count, retweet Count, like Count and view Count too.

enter image description here

Answered By: Bench Vue
Categories: questions Tags:
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.