Reddit API not giving the correct number of search results

Question

I’m trying to search for subreddits associated with a keyword by using the Reddit API but I’m having trouble trying to get 50 or more results. I’ve used the API with the requests library and now with praw and both had the same issue.

If I run this code, for most of the keywords I get the requested amount of 50 subreddits but for keywords like Entertainment, I get 48 or 49. Furthermore, if I set the limit to 100, I get 54 or 53 results for the other keywords. I tried to solve this using the after parameter, however even when the after key is assigned the fullname of the last subreddit of the previous get request, the r.get with that as a parameter returns 0 posts.

Is there any way to work around this or am I just missing something?

import praw

keywords = [
"Comedy",
"Education",
"Entertainment",
"Movies",
...
]
r = praw.Reddit(
      client_id='',
      client_secret='',
      password='',
      user_agent='',
      username='',
)

prev_completed = True
def get_data_reddit(search, after=None):
    global prev_completed
    params = {"q": search, "limit": 50}
    if after and not prev_completed:
        params["after"] = "t5_" + str(after.id)
        prev_completed = True
    posts = r.get('https://oauth.reddit.com/subreddits/search', params=params)
    if(len(posts) == 50): 
        prev_completed = True
    else:
        prev_completed = False
    return posts

post = None
count = 0
j = 0
for i in range(0, len(keywords)):
    posts = get_data_reddit(keywords[j], post)
    if len(posts) < 50:
        j -= 1
    for post in posts:
        count += 1
    j += 1
    print(count)

Asked By: Elian Malessy

||

Source

Answer 1

I realized that the parameter I was actually looking for was confusingly the before attribute, not the after attribute. With this in mind here is the code that worked for me.

prev_completed = True
postCount = 0
def get_data_reddit(search, limit, after=None,):
    global prev_completed
    global postCount
    params = {"q": search, "limit": limit}
    if after and not prev_completed:
        params["before"] = "t5_" + str(after.id)
        prev_completed = True
    posts = r.get('https://oauth.reddit.com/subreddits/search', params=params)
    postCount += len(posts)
    if(postCount == 50): 
        prev_completed = True
    else:
        prev_completed = False
    return posts

post = None
count = 0
i = 0
while i < len(keywords):
    posts = get_data_reddit(keywords[i],  50 - postCount, post)
    if postCount < 50:
        i -= 1
    else:
        postCount = 0
    for post in posts:
        count += 1
    i += 1
    print(count)

Answered By: Elian Malessy

Reddit API not giving the correct number of search results

Question:

Answers: