Click a Button in Scrapy
Question:
I’m using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).
I found out that Scrapy can handle forms (like logins) as shown here. But the problem is that there is no form to fill out, so it’s not exactly what I need.
How can I simply click a button, which then shows the information I need?
Do I have to use an external library like mechanize or lxml?
Answers:
Scrapy cannot interpret javascript.
If you absolutely must interact with the javascript on the page, you want to be using Selenium.
If using Scrapy, the solution to the problem depends on what the button is doing.
If it’s just showing content that was previously hidden, you can scrape the data without a problem, it doesn’t matter that it wouldn’t appear in the browser, the HTML is still there.
If it’s fetching the content dynamically via AJAX when the button is pressed, the best thing to do is to view the HTTP request that goes out when you press the button using a tool like Firebug. You can then just request the data directly from that URL.
Do I have to use an external library like mechanize or lxml?
If you want to interpret javascript, yes you need to use a different library, although neither of those two fit the bill. Neither of them know anything about javascript. Selenium is the way to go.
If you can give the URL of the page you’re working on scraping I can take a look.
To properly and fully use JavaScript you need a full browser engine and this is possible only with Watir/WatiN/Selenium etc.
Selenium browser provide very nice solution. Here is an example (pip install -U selenium):
from selenium import webdriver
class northshoreSpider(Spider):
name = 'xxx'
allowed_domains = ['www.example.org']
start_urls = ['https://www.example.org']
def __init__(self):
self.driver = webdriver.Firefox()
def parse(self,response):
self.driver.get('https://www.example.org/abc')
while True:
try:
next = self.driver.find_element_by_xpath('//*[@id="BTN_NEXT"]')
url = 'http://www.example.org/abcd'
yield Request(url,callback=self.parse2)
next.click()
except:
break
self.driver.close()
def parse2(self,response):
print 'you are here!'
Although it’s an old thread I’ve found quite useful to use Helium (built on top of Selenium) for this purpose and far more easier/simpler than using Selenium. It will be something like the following:
from helium import *
start_firefox('your_url')
s = S('path_to_your_button')
click(s)
...
I’m using Scrapy to crawl a webpage. Some of the information I need only pops up when you click on a certain button (of course also appears in the HTML code after clicking).
I found out that Scrapy can handle forms (like logins) as shown here. But the problem is that there is no form to fill out, so it’s not exactly what I need.
How can I simply click a button, which then shows the information I need?
Do I have to use an external library like mechanize or lxml?
Scrapy cannot interpret javascript.
If you absolutely must interact with the javascript on the page, you want to be using Selenium.
If using Scrapy, the solution to the problem depends on what the button is doing.
If it’s just showing content that was previously hidden, you can scrape the data without a problem, it doesn’t matter that it wouldn’t appear in the browser, the HTML is still there.
If it’s fetching the content dynamically via AJAX when the button is pressed, the best thing to do is to view the HTTP request that goes out when you press the button using a tool like Firebug. You can then just request the data directly from that URL.
Do I have to use an external library like mechanize or lxml?
If you want to interpret javascript, yes you need to use a different library, although neither of those two fit the bill. Neither of them know anything about javascript. Selenium is the way to go.
If you can give the URL of the page you’re working on scraping I can take a look.
To properly and fully use JavaScript you need a full browser engine and this is possible only with Watir/WatiN/Selenium etc.
Selenium browser provide very nice solution. Here is an example (pip install -U selenium):
from selenium import webdriver
class northshoreSpider(Spider):
name = 'xxx'
allowed_domains = ['www.example.org']
start_urls = ['https://www.example.org']
def __init__(self):
self.driver = webdriver.Firefox()
def parse(self,response):
self.driver.get('https://www.example.org/abc')
while True:
try:
next = self.driver.find_element_by_xpath('//*[@id="BTN_NEXT"]')
url = 'http://www.example.org/abcd'
yield Request(url,callback=self.parse2)
next.click()
except:
break
self.driver.close()
def parse2(self,response):
print 'you are here!'
Although it’s an old thread I’ve found quite useful to use Helium (built on top of Selenium) for this purpose and far more easier/simpler than using Selenium. It will be something like the following:
from helium import *
start_firefox('your_url')
s = S('path_to_your_button')
click(s)
...