Proxy Check in python

Question

I have written a script in python that uses cookies and POST/GET. I also included proxy support in my script. However, when one enters a dead proxy, the script crashes. Is there any way to check if a proxy is dead/alive before running the rest of my script?

Furthermore, I noticed that some proxies don’t handle cookies/POST headers properly. Is there any way to fix this?

Asked By: MarcoW

||

Source

Answer 1

The simplest was is to simply catch the IOError exception from urllib:

try:
    urllib.urlopen(
        "http://example.com",
        proxies={'http':'http://example.com:8080'}
    )
except IOError:
    print "Connection error! (Check proxy)"
else:
    print "All was fine"

Also, from this blog post – “check status proxy address” (with some slight improvements):

for python 2

import urllib2
import socket

def is_bad_proxy(pip):    
    try:
        proxy_handler = urllib2.ProxyHandler({'http': pip})
        opener = urllib2.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib2.install_opener(opener)
        req=urllib2.Request('http://www.example.com')  # change the URL to test here
        sock=urllib2.urlopen(req)
    except urllib2.HTTPError, e:
        print 'Error code: ', e.code
        return e.code
    except Exception, detail:
        print "ERROR:", detail
        return True
    return False

def main():
    socket.setdefaulttimeout(120)

    # two sample proxy IPs
    proxyList = ['125.76.226.9:80', '213.55.87.162:6588']

    for currentProxy in proxyList:
        if is_bad_proxy(currentProxy):
            print "Bad Proxy %s" % (currentProxy)
        else:
            print "%s is working" % (currentProxy)

if __name__ == '__main__':
    main()

for python 3

import urllib.request
import socket
import urllib.error

def is_bad_proxy(pip):    
    try:
        proxy_handler = urllib.request.ProxyHandler({'http': pip})
        opener = urllib.request.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib.request.install_opener(opener)
        req=urllib.request.Request('http://www.example.com')  # change the URL to test here
        sock=urllib.request.urlopen(req)
    except urllib.error.HTTPError as e:
        print('Error code: ', e.code)
        return e.code
    except Exception as detail:
        print("ERROR:", detail)
        return True
    return False

def main():
    socket.setdefaulttimeout(120)

    # two sample proxy IPs
    proxyList = ['125.76.226.9:80', '25.176.126.9:80']

    for currentProxy in proxyList:
        if is_bad_proxy(currentProxy):
            print("Bad Proxy %s" % (currentProxy))
        else:
            print("%s is working" % (currentProxy))

if __name__ == '__main__':
    main()

Remember this could double the time the script takes, if the proxy is down (as you will have to wait for two connection-timeouts).. Unless you specifically have to know the proxy is at fault, handling the IOError is far cleaner, simpler and quicker..

Answered By: dbr

Answer 2

I think that the better approach is like dbr said, handling the exception.

Another solution that could be better in some cases, is to use an external online proxy checker tool to check if a proxy server is alive and then continue using your script without any modification.

Answered By: Lix

Answer 3

There is one nice package Grab
So, if it ok for you, you can write something like this(simple valid proxy checker-generator):

from grab import Grab, GrabError

def get_valid_proxy(proxy_list): #format of items e.g. '128.2.198.188:3124'
    g = Grab()
    for proxy in proxy_list:
        g.setup(proxy=proxy, proxy_type='http', connect_timeout=5, timeout=5)
        try:
            g.go('google.com')
        except GrabError:
            #logging.info("Test error")
            pass
        else:
            yield proxy

Answered By: Alex Granovsky

Answer 4

you can use the Proxy-checker library which is as simple as this

from proxy_checker import ProxyChecker

checker = ProxyChecker()
checker.check_proxy('<ip>:<port>')

output :

{
  "country": "United States",
  "country_code": "US",
  "protocols": [
  "socks4",
  "socks5"
  ],
  "anonymity": "Elite",
  "timeout": 1649
}

with the possibility of generating your own proxies and check them with two lines of code

Answered By: wassim letaief

Answer 5

you can use ip-getter website to get the IP by which you are sending a request, then check if the IP is the same as your proxy IP or some thing else. Here is a script for that matter:

import requests

proxy_ip = "<IP>"
proxy_port = "<PORT>"
proxy_user = "<USERNAME>"
proxy_pass = "<PASSWORD>"

proxies = {
    "http": f"http://{proxy_user}:{proxy_pass}@{proxy_ip}:{proxy_port}/",
    "https": f"http://{proxy_user}:{proxy_pass}@{proxy_ip}:{proxy_port}/"
}

url = 'https://api.ipify.org'

try:
    response = requests.get(url, proxies=proxies)
    assert response.text==proxy_ip
except:
    print("Proxy does not work")

Answered By: Ali Cetwaty

Proxy Check in python

Question:

Answers: