Headless Browser for Python (Javascript support REQUIRED!)
Question:
I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searching for results matching certain criteria, clicking check boxes, and clicking to download files. All of this requires Javascript.
I hear a headless browser is what I want – requirements/preferences are that I be able to run it from Python, and preferably that the resultant script will be compilable by py2exe (I am writing this program for other users).
So far Windmill looks like it MIGHT be what I want, but I am not sure.
Any ideas appreciated!
Answers:
I use webkit as a headless browser in Python via pyqt / pyside:
http://www.riverbankcomputing.co.uk/software/pyqt/download
http://developer.qt.nokia.com/wiki/Category:LanguageBindings::PySide::Downloads
I particularly like webkit because it is simple to setup. For Ubuntu you just use: sudo apt-get install python-qt4
Here is an example script:
http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/
You can use HTQL in combination with IRobotSoft webscraper. Check here for examples: http://htql.net/
I’m in the midst of writing a Python driver for Zombie.js, “a lightweight framework for testing client-side JavaScript code in a simulated environment”.
I’m currently at a standstill on a resolution to a bug in Node.js (before I write more tests and more code), but feel free to keep an eye on my project as it progresses:
Try using phantomjs, it has great javascript support. Then you could run it as a subprocess of a python script
http://docs.python.org/library/subprocess.html
that could boss it around.
The answer to this question was Spynner
I need a headless browser which is fairly easy to use (I am still fairly new to Python and programming in general) which will allow me to navigate to a page, log into a form that requires Javascript, and then scrape the resulting web page by searching for results matching certain criteria, clicking check boxes, and clicking to download files. All of this requires Javascript.
I hear a headless browser is what I want – requirements/preferences are that I be able to run it from Python, and preferably that the resultant script will be compilable by py2exe (I am writing this program for other users).
So far Windmill looks like it MIGHT be what I want, but I am not sure.
Any ideas appreciated!
I use webkit as a headless browser in Python via pyqt / pyside:
http://www.riverbankcomputing.co.uk/software/pyqt/download
http://developer.qt.nokia.com/wiki/Category:LanguageBindings::PySide::Downloads
I particularly like webkit because it is simple to setup. For Ubuntu you just use: sudo apt-get install python-qt4
Here is an example script:
http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/
You can use HTQL in combination with IRobotSoft webscraper. Check here for examples: http://htql.net/
I’m in the midst of writing a Python driver for Zombie.js, “a lightweight framework for testing client-side JavaScript code in a simulated environment”.
I’m currently at a standstill on a resolution to a bug in Node.js (before I write more tests and more code), but feel free to keep an eye on my project as it progresses:
Try using phantomjs, it has great javascript support. Then you could run it as a subprocess of a python script
http://docs.python.org/library/subprocess.html
that could boss it around.
The answer to this question was Spynner