I am trying to extract data from class using selenium but it is not working

Question

I need to scrape data from this website:
https://www.daraz.pk/products/hy-i189662857-s1379994759.html?spm=a2a0e.searchlistcategory.list.3.70426378Fs3yJh&search=1

and html code for this is:

<div class="mod-reviews">
   <div class="item">...</div>
   <div class="item">...</div>
   <div class="item">...</div>
   <div class="item">...</div>
</div>

Each class div has 1 review and i want to extract it.
I tried following code with selenium:

driver.get(site)
time.sleep(5)
data = driver.find_elements(By.CLASS_NAME,"item")

Hypothetically this should give me all item tags but for some reason this returns nothing. Am I missing something here?

Asked By: HAPPY LAMMA

||

Source

Answer 1

These elements only appear after you scroll down the page for a while.
Try to GUI-Automate that with a javascript injection like this

for i in range(25):
    driver.execute_script("window.scrollBy(0,100)")

Sadly, that scrolling does not work until you put your mouse cursor there, so there will other gui automation be necessary as well.

But wait! you can trick it:

for i in range(10):
    driver.execute_script("window.scrollBy(0,100)")
for i in range(10):
    driver.execute_script("window.scrollBy(0,-100)")
for i in range(25):
    driver.execute_script("window.scrollBy(0,100)")

Answered By: Ohnemichel

I am trying to extract data from class using selenium but it is not working

Question:

Answers: