Python Max retries exceeded with url while using request

Question

I was working on urls using request library. so I can check that if the URL is working or not using get method. The Script works fine for the rest of URLs but for one url it took a lot of time and then this error occur:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='macromedia.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000001E8ACACF040>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))

Now all I want to skip url when timeout occur. I tried continue pass so that somehow I move forwad but it still fails below is the code:

import time

import requests
import bs4
from Base_Class import *
import threading

class Checking_Valid_URL:

    def __init__(self):
        self.https = 0
        self.http = 0
        #database connection
        #print(urls)
        self.url_protocols = ['http://', 'https://']
        #database connection

    def Checking_for_http_https_content_status(self):
        for url in self.urls:
            for url_protocol in self.url_protocols:
                try:
                    time.sleep(2)
                    full_https_url = url_protocol + url[0]
                    res = requests.get(full_https_url, timeout=60)
                    soup = bs4.BeautifulSoup(res.text, 'html.parser')
                    elems = soup.select('body')
                    try:
                        if elems:
                            print(f'body found in {full_https_url}')
                            try:
                                if res.status_code == 200:
                                    #database connection
                                    if full_https_url.startswith('https'):
                                        print('https:: ' + full_https_url + ' ' + str(res.status_code))
                                        try:
                                            #database connection
                                            self.https += 1
                                            time.sleep(5)
                                        except:
                                            continue

                                    elif full_https_url.startswith('http'):

                                        print('https:: ' + full_https_url + ' ' + str(res.status_code))
                                        try:
                                            #database connection
                                            self.http += 1
                                            time.sleep(5)
                                        except:
                                            continue
                            except:
                                continue

                        else:
                            print(f"No body in {full_https_url}")
                            continue
                    except:
                        print(f"No body in {full_https_url}")
                        continue

                except requests.exceptions.Timeout:
                    print(f"Timeout on {full_https_url}, skipping")
                    continue

check = Checking_Valid_URL()
check.Checking_for_http_https_content_status()

the base class has database creation and selenium driver nothing else. #database connection data comming from data base so i removed it the line of code will exceed too much. but it’s working.

Asked By: stack overflow

||

Source

Answer 1

Are you actually experiencing a Timeout exception now or still an exception of class requests.exceptions.ConnectionError ? (if the latter this would explain why your current exception handling code isn’t working).

You should be catch both ConnectionError and Timeout exceptions along with a generic catch all.

For starters, just try

except requests.exceptions.RequestException as e:
    # some sort of connection error
    continue

Answered By: Ganondorfz

Python Max retries exceeded with url while using request

Question:

Answers: