Unable to Click to next page while crawling selenium

Question:

I’m trying to scrap the list of services we have for us from this site but not able to click to the next page.
This is what I’ve tried so far using selenium & bs4,

#attempt1

next_pg_btn = browser.find_elements(By.CLASS_NAME, 'ui-lib-pagination_item_nav')
next_pg_btn.click() # nothing happens

#attemp2

browser.find_element(By.XPATH, "//div[@role = 'button']").click() # nothing happens

#attempt3 - saw in some stackoverflow post that sometimes we need to scroll to the 
#bottom of page to have the button clickable, so tried that

browser.execute_script("window.scrollTo(0,2500)")
browser.find_element(By.XPATH, "//div[@role = 'button']").click() # nothing happens

I’m not so experienced with scrapping, pls advice how to handle this and where I’m going wrong.
Thanks

Asked By: Aman Singh

||
Source

Answers:

Several issues with your code:

  1. You tried wrong locators.
  2. You probably need to wait for the element to be loaded before clicking it. But if before clicking the pagination you performing some actions on the page this is not needed since during you scraping the page content web elements are already got loaded.
  3. Pagination button is on the buttom of the page, so you need to scroll the page to bring the pagination button into the visible screen.
  4. After scrolling some delay should be added, as you can see in the code below.
  5. Now pagination element can be clicked.

The following code works

import time

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:webdriverschromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)

url = "https://www.tamm.abudhabi/en/life-events/individual/HousingProperties"
driver.get(url)
pagination = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "ui-lib-pagination__item_nav")))
pagination.location_once_scrolled_into_view
time.sleep(0.5)
pagination.click()
Answered By: Prophet