Turning on debug output for python 3 urllib

Question:

In python 2, it was possible to get debug output from urllib by doing

import httplib
import urllib
httplib.HTTPConnection.debuglevel = 1
response = urllib.urlopen('http://example.com').read()

However, in python 3 it looks like this has been moved to 

http.client.HTTPConnection.set_debuglevel(level)

However, I’m using urllib not http.client directly. How can I set it up so that my http request display debugging information in this way?

Here’s what I”m using so far. What’s the best way to proceed if I want to be able to get debug information?

#Request Login page
cookiejar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookiejar))
request = urllib.request.Request(options.uri)
add_std_headers(request)
response = opener.open(request)
response_string = response.read().decode("utf8")
# ...
Asked By: Zxaos

||
Source

Answers:

You were right the first time. You can simply add the line http.client.HTTPConnection.debuglevel = 1 at the start of your file to turn on HTTP debugging application-wide. urllib.request still uses http.client.

It seems that there’s also a way to set the debuglevel for a single handler (by creating urllib.request.HTTPHandler(debuglevel=1) and building an opener with that), but on my installation of Python3 (3.0b3) it’s not actually implemented. I imagine that’s changed in more recent versions!

Answered By: PAG

For those who are seeing this question more recently (since ~May, 2016), while the accepted answer might have been correct at some point, it appears that since Python version 3.5.2 the http.client.HTTPConnection.debuglevel is entirely ignored in favor of the debuglevel constructor argument for urllib.request.HTTPHandler.

This is due to this change that sets the value of http.client.HTTPConnection.debuglevel to whatever is set in urllib.request.HTTPHandler‘s constructor argument debuglevel, on this line

A PR has been opened to fix this, but in the mean time you can monkey patch the __init__ methods of HTTPHandler and HTTPSHandler to respect the global values like so:

https_old_init = urllib.request.HTTPSHandler.__init__

def https_new_init(self, debuglevel=None, context=None, check_hostname=None):
    debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
    https_old_init(self, debuglevel, context, check_hostname)

urllib.request.HTTPSHandler.__init__ = https_new_init

http_old_init = urllib.request.HTTPHandler.__init__

def http_new_init(self, debuglevel=None):
    debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
    http_old_init(self, debuglevel)

urllib.request.HTTPHandler.__init__ = http_new_init

Note: I don’t recommend setting the debuglevel in HTTPHandler‘s as a method argument default value because the default values for method arguments get evaluated at function definition evaluation time, which, for HTTPHandler‘s constructor, is when the module urllib.request is imported.

Answered By: wheeler

It looks like it is different today than when this was last answered.

My default urllib3 connections are going thru PoolManager.

Turning on debug level is somewhat missing, but it does have a nice logger usage now.

https://urllib3.readthedocs.io/en/stable/user-guide.html#logging

values go from

  • DEBUG
  • INFO
  • WARNING
  • ERROR
  • CRITICAL

Here is the line to start sharing the debug info to stdout

import logging
logging.basicConfig(format='%(asctime)s - %(message)s', level=logging.DEBUG)
logging.getLogger("urllib3").setLevel(logging.DEBUG)

This will give you output like this:

2023-02-01 21:02:24,221 - Resetting dropped connection: localhost
2023-02-01 21:02:24,227 - http://localhost:9000 "GET /web/path HTTP/1.1" 200 100
Answered By: phyatt