How to retry urllib2.request when fails?

Question

When urllib2.request reaches timeout, a urllib2.URLError exception is raised.
What is the pythonic way to retry establishing a connection?

Asked By: iTayb

||

Source

Answer 1

I would use a retry decorator. There are other ones out there, but this one works pretty well. Here’s how you can use it:

@retry(urllib2.URLError, tries=4, delay=3, backoff=2)
def urlopen_with_retry():
    return urllib2.urlopen("http://example.com")

This will retry the function if URLError is raised. Check the link above for documentation on the parameters, but basically it will retry a maximum of 4 times, with an exponential backoff delay doubling each time, e.g. 3 seconds, 6 seconds, 12 seconds.

Answered By: jterrace

Answer 2

To retry on timeout you could catch the exception as @Karl Barker suggested in the comment:

assert ntries >= 1
for i in range(1, ntries+1):
    try:
        page = urlopen(request, timeout=timeout)
        break # success
    except URLError as err:
        if i == ntries or not isinstance(err.reason, socket.timeout):
           raise # propagate last timeout or non-timeout errors
# use page here

Answered By: jfs

Answer 3

There are a few libraries out there that specialize in this.

One is backoff, which is designed with a particularly functional sensibility. Decorators are passed arbitrary callables returning generators which yield successive delay values. A simple exponential backoff with a maximum retry time of 32 seconds could be defined as:

@backoff.on_exception(backoff.expo,
                      urllib2.URLError,
                      max_value=32)
def url_open(url):
    return urllib2.urlopen("http://example.com")

Another is retrying which has very similar functionality but an API where retry parameters are specified by way of predefined keyword args.

Answered By: bgreen-litl

Answer 4

For Python3, you can use urllib3.Retry:

from urllib3 import Retry, PoolManager


retries = Retry(connect=5, read=2, redirect=5, backoff_factor=0.1)
http = PoolManager(retries=retries)
response = http.request('GET', 'http://example.com/')

If the backoff_factor is 0.1, then :func:.sleep will sleep
for [0.0s, 0.2s, 0.4s, …] between retries. It will never be longer
than :attr:Retry.BACKOFF_MAX.
urllib3 will sleep for::
        {backoff factor} * (2 ** ({number of total retries} - 1))

Answered By: LinPy

How to retry urllib2.request when fails?

Question:

Answers: