How can I use XPATH to search WEB elements in pageSource constraint: without using BeautifulSoup only XPATH

Question:

I made a program to scrape data from a website in which I search elements by using XPath, e.g.

url is a Book site containing book covers on main page and clicking on book covers redirect us to new webpage which has data about single book

driver.get(url)
title = driver.find_elements(By.XPATH,'//div[@class="title"]').text

When I have greater amount of links this process is very slow, so is there some way that I can find these tag elements using page_source?

driver.get(url)
content = driver.page_source
title = content.find_elements(By.XPATH,'//div[@class="title"]')

However this does not work because the content is a string, and as such has no method find_elements

I don’t want to use Beautiful Soup to find elements using class names and their attributes, I want to find element by using XPATH.

Is it Possible to do this?

Asked By: lokp

||
Source

Answers:

Selnium does not provide any way to use XPATH on page_cource stored in a variable.Either if you want to do it use lxml.etree to do it.
lxml.etree tutorial

this will help you to use lxml.etree to use xpath on page_source

Answered By: poliop