Max retries exceeded with url error Python Requests

Question

I’m trying to connect to a website with the code below. There is no problem on Heroku. But I am getting error in DigitalOcean.

Code:

headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"}

def web():
  rq = requests.get("https://myurl.com/gts?search=word", headers = headers)
  print(rq)

The error I get in DigitalOcean:

HTTPSConnectionPool(host='myurl.com', port=443): Max retries exceeded with url: /gts?search=word (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa364a89690>, 'Connection to myurl.com timed out. (connect timeout=None)'))

What I’ve done and failed:

I added code verify=False, disabled firewall from console. And I tried to access the site from the console with the "curl -I myurl.com" command. All failed.

Thank you for your help.

Asked By: Emir

||

Source

Answer 1

It seems like the issue is not caused by a bug in your code, but rather by firewall (or other rules) on the remote server side (sozluk.gov.tr).

I have tried to run curl -I https://sozluk.gov.tr and also your snippet on my local and also remote (Digital Ocean hosted) VM – both worked fine.

You mentioned that curl command from console (I assume remove VM console) did not work. This indicates, that the issue is on the network – rather than code – level.

I recommend to spin new VM in different region (to get IP from different IP pool) or use some kind of a proxy which is not blocked by remote server. You can check available regions (datacenters) on this link.

Responses

user@do-server:~$ curl -I https://sozluk.gov.tr
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 18 Feb 2023 11:18:37 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 108975
Last-Modified: Fri, 13 Jan 2023 12:38:15 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "63c150b7-1a9af"
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, OPTIONS
Accept-Ranges: bytes

user@do-server:~$ python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36"}
>>> requests.get("https://sozluk.gov.tr/gts?search=word", headers = headers)
<Response [200]>

Answered By: FN_

Max retries exceeded with url error Python Requests

Question:

Answers: