Python Reddit API JSON issues (no PRAW)

Question:

I am trying to obtain replies to the comments on Threads. Here is what I have been able to accomplish by parsing JSON:

subreddit =  'wallstreetbets'
link = 'https://oauth.reddit.com/r/'+subreddit+'/hot'
hot = requests.get(link,headers = headers)
hot.json()

Here is output

{'kind': 'Listing',
 'data': {'after': 't3_x8kidp',
  'dist': 27,
  'modhash': None,
  'geo_filter': None,
  'children': [{'kind': 't3',
    'data': {'approved_at_utc': None,
     'subreddit': 'wallstreetbets',
     'selftext': '**Read [rules](https://www.reddit.com/r/wallstreetbets/wiki/contentguide), follow [Twitter](https://twitter.com/Official_WSB) and [IG](https://www.instagram.com/official_wallstreetbets/), join [Discord](https://discord.gg/wallstreetbets), see [ban bets](https://www.reddit.com/r/wallstreetbets/wiki/banbets)!**nn[dm mods because why not](https://www.reddit.com/message/compose/?to=/r/wallstreetbets)nn[Earnings Thread](https://wallstreetbets.reddit.com/x4ryjg)',
     'author_fullname': 't2_bd6q5',
     'saved': False,
     'mod_reason_title': None,
     'gilded': 0,
     'clicked': False,
     'title': 'What Are Your Moves Tomorrow, September 08, 2022',
     'link_flair_richtext': [{'e': 'text', 't': 'Daily Discussion'}],
     'subreddit_name_prefixed': 'r/wallstreetbets',
     'hidden': False,
     'pwls': 7,
     'link_flair_css_class': 'daily',
     'downs': 0,
     'thumbnail_height': None,
     'top_awarded_type': None,
     'hide_score': False,
     'name': 't3_x8ev67',
...
     'created_utc': 1662594703.0,
     'num_crossposts': 0,
     'media': None,
     'is_video': False}}],
  'before': None}}

I then turned it into a data frame

df = pd.DataFrame()
for post in hot.json()['data']['children']:
    df = df.append({
        'subreddit' : post['data']['subreddit'],
        'title': post['data']['title'],
        'selftext': post['data']['selftext'],
        'created_utc': post['data']['created_utc'],
        'id': post['data']['id']

      

    }, ignore_index = True)

With this, I was able to obtain a data frame like thisDataFrame

Then, to obtain the comments, I created a list with all the JSON script from the 26 posts, and then created a while loop to iterate through the json script.

supereme = len(list_of_comments)
indexy = pd.DataFrame()
while supereme > 0:
    supereme -= 1
    for g in range(0,len(list_of_comments[supereme]['data']['children'])-1):
        indexy = pd.concat([indexy, pd.DataFrame.from_records([{
     'body': list_of_comments[supereme]['data']['children'][g]['data']['body'],
     'post_id': list_of_comments[supereme]['data']['children'][g]['data']['parent_id'] }])], ignore_index = True)

      

  
indexy

This gave me this: DataFrame
However, I am not able to obtain the replies to the comments. Any help? I tried to do this

posts = 26 
for i in np.arange(0,27):
    print('i',i)
    if len(list_of_comments[i]['data']['children']) == 0:
        continue
    for j in np.arange(0,len(list_of_comments[i]['data']['children'])):
        if len(list_of_comments[i]['data']['children'][j]['data']['replies']) == 0:
            break
        else: 
            print('j',len(list_of_comments[i]['data']['children'][j]['data']['replies']))
            for z in np.arange(len(list_of_comments[i]['data']['children'][j]['data']['replies']['data']['children'])):
                if len(list_of_comments[i]['data']['children'][j]['data']['replies']['data']['children']) == 0:
                    break
                print('z',z)


                print(list_of_comments[i]['data']['children'][j]['data']['replies']['data']['children'][z]['data']['body'])

The first loop kinda works but it doesn’t count up properly to get all the replies to all the posts itll only pull like one or two. We don’t want to use PRAW

Asked By: user19890024

||

Answers:

x=len(list_of_comments)
replies = pd.DataFrame()
for i in range(0,len(list_of_comments)):
    try:
        for j in range(0, len(list_of_comments[x]['data']['children'])):  
            try: 
                for z in range(0, len(list_of_comments[x]['data']['children'][j]['data']['replies']['data']['children'])):
                    #print(list_of_comments[x]['data']['children'][j]['data']['replies']['data']['children'][z]['data']['body'])
                    #print(list_of_comments[x]['data']['children'][j]['data']['replies']['data']['children'][z]['data']['link_id'])
                    replies = pd.concat([replies, pd.DataFrame.from_records([{
        'body': list_of_comments[x]['data']['children'][j]['data']['replies']['data']['children'][z]['data']['body'],
        'post_id': list_of_comments[x]['data']['children'][j]['data']['replies']['data']['children'][z]['data']['link_id']

      

    }])], ignore_index = True)
                    
            except:
                pass
    except:
        continue
Answered By: Ish A
Categories: questions Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.