How to add random delays between the queries sent to Google to avoid getting blocked in python
Question:
I have written a program which sends more than 15 queries to Google in each iteration, total iterations is about 50. For testing I have to run this program several times. However, by doing that, after several times, Google blocks me. is there any ways so I can fool google maybe by adding delays between each iteration? Also I have heard that google can actually learn the timesteps. so I need these delays to be random so google cannot find a patter from it to learn my behavior. also it should be short so the whole process doesn’t take so much.
Does anyone know something, or can provide me a piece of code in python?
Answers:
First, Google probably are blocking you because they don’t like it when you take too many of their resources. The best way to fix this is to slow it down, not delay randomly. Stick a 1 second wait after every request and you’ll probably stop having problems.
That said:
from random import randint
from time import sleep
sleep(randint(10,100))
will sleep a random number of seconds (between 10 and 100).
Since you’re not testing Google’s speed, figure out some way to simulate it when doing your testing (as @bstpierre suggested in his comment). This should solve your problem and factor its variable response times out at the same time.
Also you can try to use few proxy servers for prevent ban by IP adress. urllib support proxies by special constructor parameter, httplib can use proxy too
Best to use:
from numpy import random
from time import sleep
sleeptime = random.uniform(2, 4)
print("sleeping for:", sleeptime, "seconds")
sleep(sleeptime)
print("sleeping is over")
as a start and slowly decreasy range to see what works best (fastest).
For anyone stumbling here for the general "how to add random delay to my routine" case in 2022, numpy’s recommended method [1] is to use their random number generator class:
from numpy.random import default_rng
from time import sleep
rng = default_rng()
# generates a scalar [single] value greater than or equal to 1
# but less than 3
time_to_sleep = rng.uniform(1, 3)
sleep(time_to_sleep)
[1] https://numpy.org/doc/stable/reference/random/index.html#quick-start
I have written a program which sends more than 15 queries to Google in each iteration, total iterations is about 50. For testing I have to run this program several times. However, by doing that, after several times, Google blocks me. is there any ways so I can fool google maybe by adding delays between each iteration? Also I have heard that google can actually learn the timesteps. so I need these delays to be random so google cannot find a patter from it to learn my behavior. also it should be short so the whole process doesn’t take so much.
Does anyone know something, or can provide me a piece of code in python?
First, Google probably are blocking you because they don’t like it when you take too many of their resources. The best way to fix this is to slow it down, not delay randomly. Stick a 1 second wait after every request and you’ll probably stop having problems.
That said:
from random import randint
from time import sleep
sleep(randint(10,100))
will sleep a random number of seconds (between 10 and 100).
Since you’re not testing Google’s speed, figure out some way to simulate it when doing your testing (as @bstpierre suggested in his comment). This should solve your problem and factor its variable response times out at the same time.
Also you can try to use few proxy servers for prevent ban by IP adress. urllib support proxies by special constructor parameter, httplib can use proxy too
Best to use:
from numpy import random
from time import sleep
sleeptime = random.uniform(2, 4)
print("sleeping for:", sleeptime, "seconds")
sleep(sleeptime)
print("sleeping is over")
as a start and slowly decreasy range to see what works best (fastest).
For anyone stumbling here for the general "how to add random delay to my routine" case in 2022, numpy’s recommended method [1] is to use their random number generator class:
from numpy.random import default_rng
from time import sleep
rng = default_rng()
# generates a scalar [single] value greater than or equal to 1
# but less than 3
time_to_sleep = rng.uniform(1, 3)
sleep(time_to_sleep)
[1] https://numpy.org/doc/stable/reference/random/index.html#quick-start