Infinite scrolling of webpage using Selenium Python

Question

I want to scroll till all the element is loaded. My code works well for starting till 7 8 pages using this code.

driver.get('https://www.bigbasket.com/ps/?q=rice')
last_height = driver.execute_script("return document.body.scrollHeight")

while True:
    # Scroll down to bottom
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # Wait to load page
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//div[@class="show-more"]/button')))
    driver.find_element_by_xpath('//div[@class="show-more"]/button').click()
    # Calculate new scroll height and compare with last scroll height
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

The last height became equal to new height even though not all the element is loaded. I need to scroll till all element is loaded.

Asked By: Salman Ansari

||

Source

Answer 1

This might help you out as it has several suggestions as to how to do scrolling and how to get true height: https://javascript.info/size-and-scroll-window

Answered By: Sergey Sarnavskiy

Answer 2

To scroll down till the bottom of the page and keep on clicking the Show More button you can use the following locator strategy:

Code block:

driver.get("https://www.bigbasket.com/ps/?q=rice")
while True:
  try:
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
      print("Browser scrolled")
      driver.execute_script("arguments[0].click();", WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Show More']"))))
      print("Show More button clicked")
      time.sleep(3)
  except WebDriverException:
      print("No more scrolling")
      break

Console Output:

Browser scrolled
Show More button clicked
Browser scrolled
Show More button clicked
Browser scrolled
Show More button clicked
...
...

Answered By: undetected Selenium

Infinite scrolling of webpage using Selenium Python

Question:

Answers: