Which is best in Python: urllib2, PycURL or mechanize?

I think this talk (at pycon 2009), has the answers for what you’re looking for (Asheesh Laroia has lots of experience on the matter). And he points out the good and the bad from most of your listing

• Scrape the Web: Strategies for
  programming websites that don’t
  expect it (Part 1 of 3)
• Scrape the Web: Strategies for
  programming websites that don’t
  expect it (Part 2 of 3)
• Scrape the Web: Strategies for
  programming websites that don’t
  expect it (Part 3 of 3)

From the PYCON 2009 schedule:

Do you find yourself faced with
websites that have data you need to
extract?
Would your life be simpler if
you could programmatically input data
into web applications, even those
tuned to resist interaction by bots?

We’ll discuss the basics of web
scraping, and then dive into the
details of different methods and where
they are most applicable.

You’ll leave
with an understanding of when to apply
different tools, and learn about a
“heavy hammer” for screen scraping
that I picked up at a project for the
Electronic Frontier Foundation.

Atendees should bring a laptop, if
possible, to try the examples we
discuss and optionally take notes.

Update:
Asheesh Laroia has updated his presentation for pycon 2010

• PyCon 2010: Scrape the Web:
  Strategies for programming websites
  that don’t expected it

  * My motto: "The website is the API."
  * Choosing a parser: BeautifulSoup, lxml, HTMLParse, and html5lib.
  * Extracting information, even in the face of bad HTML: Regular expressions, BeautifulSoup, SAX, and XPath.
  * Automatic template reverse-engineering tools.
  * Submitting to forms.
  * Playing with XML-RPC
  * DO NOT BECOME AN EVIL COMMENT SPAMMER.
  * Countermeasures, and circumventing them:
        o IP address limits
        o Hidden form fields
        o User-agent detection
        o JavaScript
        o CAPTCHAs 
  * Plenty of full source code to working examples:
        o Submitting to forms for text-to-speech.
        o Downloading music from web stores.
        o Automating Firefox with Selenium RC to navigate a pure-JavaScript service. 
  * Q&A; and workshopping
  * Use your power for good, not evil. 
  

Update 2:

PyCon US 2012 – Web scraping: Reliably and efficiently pull data from pages that don’t expect it

Exciting information is trapped in web pages and behind HTML forms. In this tutorial, >you’ll learn how to parse those pages and when to apply advanced techniques that make >scraping faster and more stable. We’ll cover parallel downloading with Twisted, gevent, >and others; analyzing sites behind SSL; driving JavaScript-y sites with Selenium; and >evading common anti-scraping techniques.

Don’t worry about “last updated”. HTTP hasn’t changed much in the last few years 😉

urllib2 is best (as it’s inbuilt), then switch to mechanize if you need cookies from Firefox. mechanize can be used as a drop-in replacement for urllib2 – they have similar methods etc. Using Firefox cookies means you can get things from sites (like say StackOverflow) using your personal login credentials. Just be responsible with your number of requests (or you’ll get blocked).

PycURL is for people who need all the low level stuff in libcurl. I would try the other libraries first.

Urllib2 only supports HTTP GET and POST, there might be workarounds, but If your app depends on other HTTP verbs, you will probably prefer a different module.

Every python library that speaks HTTP has its own advantages.

Use the one that has the minimum amount of features necessary for a particular task.

Your list is missing at least urllib3 – a cool third party HTTP library which can reuse a HTTP connection, thus speeding up greatly the process of retrieving multiple URLs from the same site.

Python requests is also a good candidate for HTTP stuff. It has a nicer api IMHO, an example http request from their offcial documentation:

>>> r = requests.get('https://api.github.com', auth=('user', 'pass'))
>>> r.status_code
204
>>> r.headers['content-type']
'application/json'
>>> r.content
...

Take a look on Grab (http://grablib.org). It is a network library which provides two main interfaces:
1) Grab for creating network requests and parsing retrieved data
2) Spider for creating bulk site scrapers

Under the hood Grab uses pycurl and lxml but it is possible to use other network transports (for example, requests library). Requests transport is not well tested yet.

To “get some webpages”, use requests!

From http://docs.python-requests.org/en/latest/ :

Python’s standard urllib2 module provides most of the HTTP
capabilities you need, but the API is thoroughly broken. It was built
for a different time — and a different web. It requires an enormous
amount of work (even method overrides) to perform the simplest of
tasks.

Things shouldn’t be this way. Not in Python.

>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}