Pytrends ResponseErrore code 429 for a search list of many words

Question

I am trying to get google search volume index using Pytrends. However, when I use a large set of words, I get the following error:

TooManyRequestsError: The request failed: Google returned a response with code 429

There are similar posts on this issue (for example here). However, none of the solutions in this step work in my case. I thought maybe there is a way to introduce a certain stop in time after several searches? (in total I have 12000 words). When I run the code with a smaller number of words in a list, the code works fine. Below is the code:

from pytrends.request import TrendReq
import pandas as pd 

pytrends = TrendReq()

terms_df = pd.read_csv('search_words.csv', sep=',', low_memory=False)
terms = list(terms_df['words'])


def scrape_google(term):
    pytrends.build_payload(term, cat=0, timeframe='all', geo='US', gprop='')
    trends = pytrends.interest_over_time()
    try:
        trends = trends.drop(columns=['isPartial'])
    except:
        pass

    return trends

def get_trends(terms):
    for i in range(0,len(terms)):
        if i == 0:
            trends = scrape_google(terms[i:i+1])
        else:
            trends = pd.concat([trends, scrape_google(terms[i:i+1])], axis=1)
    return trends

trends = get_trends(terms)

Alternative way:

from pytrends.request import TrendReq
import pandas as pd

pytrends = TrendReq(hl='en-US', tz=360, timeout=(10, 25))


pytrends.build_payload(terms, cat=0, timeframe='all', geo='US', gprop='')
trends = pytrends.interest_over_time()
trends = trends.drop(columns=['isPartial'])

Here I dont know how to make a loop that will go over a list (terms) and take just 5 words (maximum in pytrends) and that will concatenate pd DF into one.

Asked By: Alberto Alvarez

||

Source

Answer 1

Here’s how you would do 5 at a time. The key is to run your loop in increments of 5, not increments of 1.

def get_trends(terms):
    trends = scrape_google(terms[0:5])
    for i in range(5,len(terms),5):
        trends = pd.concat([trends, scrape_google(terms[i:i+5])], axis=1)
    return trends

Answered By: Tim Roberts

Pytrends ResponseErrore code 429 for a search list of many words

Question:

Answers: