Looking for alternative of scrapy.Selector in Selenium project
Question:
Right now using scrapy.Selector to extract data from driver.page_source (Selenium). Looking for another way of doing this without loading scrapy library. Don’t want to use driver.find_elements method
import selenium, scrapy
from scrapy import Selector
driver.get(link)
page_source = driver.page_source
selector = Selector(text=page_source)
links = selector.xpath('//a[contains(@class, "jcs-JobTitle")]/@href').extract()
next_page = selector.xpath('//a[@aria-label="Next Page"]/@href').extract_first()
Answers:
Use parsel
parsel is the scrapy selector library, just without the rest of scrapy.
The only part of your code that would need changing is the imports. you also might have to use get and getall instead of extract_first and extract.
import selenium
from parsel import Selector
driver.get(link)
page_source = driver.page_source
selector = Selector(text=page_source)
links = selector.xpath('//a[contains(@class, "jcs-JobTitle")]/@href').getall()
next_page = selector.xpath('//a[@aria-label="Next Page"]/@href').get()
Right now using scrapy.Selector to extract data from driver.page_source (Selenium). Looking for another way of doing this without loading scrapy library. Don’t want to use driver.find_elements method
import selenium, scrapy
from scrapy import Selector
driver.get(link)
page_source = driver.page_source
selector = Selector(text=page_source)
links = selector.xpath('//a[contains(@class, "jcs-JobTitle")]/@href').extract()
next_page = selector.xpath('//a[@aria-label="Next Page"]/@href').extract_first()
Use parsel
parsel is the scrapy selector library, just without the rest of scrapy.
The only part of your code that would need changing is the imports. you also might have to use get and getall instead of extract_first and extract.
import selenium
from parsel import Selector
driver.get(link)
page_source = driver.page_source
selector = Selector(text=page_source)
links = selector.xpath('//a[contains(@class, "jcs-JobTitle")]/@href').getall()
next_page = selector.xpath('//a[@aria-label="Next Page"]/@href').get()