cannot click load more button with selenium

Question:

I’m trying to click load more button several times using selenium, however I cannot click load more button (it is even not a button…)
When I try to click it, it shows error element click intercepted: Element is not clickable at point even after I explicitly code wait.until(EC.element_to_be_clickable)

I wonder what can be wrong with my code. The code I’m using is

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
import time
url = "https://www.architectural-review.com/buildings/house"
    
options = webdriver.ChromeOptions() 
driver = webdriver.Chrome()
driver.get(url)
    
# accept cookies
wait0 = WebDriverWait(driver, 10)
cookie_button = wait0.until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[1]/div/div[6]/button[1]')))
cookie_button.click()
    
print("loading more projects ...")
count=5
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.XPATH,'/html/body/div[2]/div/div[3]/section[5]/div/div[5]/a')))
print(element)
print("now element is clickable")
while count>1:
   element.click();
   count-=1
   print(count)

and the HTML looks like (it’s not a button, not sure whether this matters)

<div class="view-more">
    <a href="#" class="cpb-bottom-more-news-link dynamic-loader" data-term-id="869" data-block-id="7" data-offset="18" data-load-number="4" data-resulting-class="fourth-post" data-exclude="MzE4Mzk5LDMxODM2MCwzMTc2MDAsNjUyNjEsNjQzMTMsNjQzMTgsNjQyNzMsNjMyNTc=">Load More </a>
</div>
Asked By: icarus.z

||
Source

Answers:

This is somewhat tricky.
The "Load more" button is initially out of the screen here, so you need to scroll the page in order to click that button. But when you scroll this page it scrolls more than needed so the "Load more" button appears out of the screen again, so this time you will have to scroll back 🙂
"Load more" button presents on the page even when nothing to load more. In this case this button just becomes invisible. So, to stop scrolling you need to check style attribute of that element.
Also, you need to improve your locators. Very long absolute XPaths are extremely breakable and not reliable.
The following code works!

import time

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:webdriverschromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)

url = "https://www.architectural-review.com/buildings/house"
driver.get(url)

wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.cmplz-btn.cmplz-accept'))).click()
while True:
    load_more = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "cpb-bottom-more-news-link")))
    load_more.location_once_scrolled_into_view
    time.sleep(0.4)
    driver.execute_script("window.scrollBy(0, arguments[0]);", -300)
    time.sleep(0.4)
    displayed = load_more.get_attribute("style")
    if "none" in displayed:
        break
    driver.execute_script("arguments[0].click();", load_more)
Answered By: Prophet

Here is a way of getting that information without Selenium (which is primarily a testing tool, and should be the last call for a web scraping venture, when all else fails):

import requests
from bs4 import BeautifulSoup as bs
import base64
from tqdm import tqdm ## if using Jupyter, do `from tqdm.notebook import tqdm`
import pandas as pd

headers = {
    'referer': 'https://www.architectural-review.com/buildings/house',
    'x-requested-with': 'XMLHttpRequest',
    'accept-language': 'en-US,en;q=0.9',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}
s = requests.Session()
s.headers.update(headers)
big_list = []
for x in tqdm(range(0, 50, 4)):
    url = f'https://www.architectural-review.com/wp-admin/admin-ajax.php?action=lazy-load&offset={x}&term_id=869&block_id=7&class=fourth-post&load_number=4&exclude=MzE4Mzk5LDMxODM2MCwzMTc2MDAsNjUyNjEsNjQzMTMsNjQzMTgsNjQyNzMsNjMyNTcsMzI2NzE2LDMyNjY5OSwzNDY3MSwzMjg3MzgsMTc5ODY1LDE3OTgzNSwxNzk3NjgsMTc5NTk4LDg3NDgyLDY3Mjk2LDY1NjIwLDYzNjQwLDYzMzgyLDYzMzQzLDYzMjk3LDYyOTMxLDYyOTI1LDYyMTA2LDYwOTE4LDU3MjMyLDU3MTE0LDU3MTAzLDU3MDg4LDU3MDgxLDU3MDc1LDU2OTkzLDU2OTc4'
    r = s.get(url)
    soup = bs(base64.b64decode(r.json()['body']), 'html.parser')
    houses = soup.select('div[class="lazy-post fourth-post"]')
    for house in houses:
        title = house.select_one('h2').get_text(strip=True)
        author = house.select_one('span[class="tie-author"]').get_text(strip=True)
        url = house.select_one('h2').parent.get('href')
        big_list.append((title, author, url))
df = pd.DataFrame(big_list, columns=['title', 'author', 'url'])
print(df)

Result in terminal:

100%
13/13 [00:13<00:00, 1.10it/s]
title   author  url
0   Hlöðuberg artist studio in Skarðsströnd, Iceland by Studio Bua  Olivia Wynne Houck  https://www.architectural-review.com/awards/ar-house/hloduberg-artist-studio-in-skardsstrond-iceland-by-studio-bua
1   Weekend house on Salamis Island, Greece by AREA (Architecture Research Athens)  Dorette Panagiotopoulou https://www.architectural-review.com/awards/ar-house/weekend-house-on-salamis-island-greece-by-area-architecture-research-athens
2   Beaconsfield House in Perth, Australia by Simon Pendal Architect    Rachael Bernstone   https://www.architectural-review.com/awards/ar-house/beaconsfield-house-in-perth-australia-by-simon-pendal-architect
3   House in Kanazawa in Ishikawa Prefecture, Japan by Shota Nakanishi Architects   Danielle Demetriou  https://www.architectural-review.com/awards/ar-house/house-in-kanazawa-in-ishikawa-prefecture-japan-by-shota-nakanishi-architects
4   House on Hamburgö in BohĂŒslan, Sweden by Manthey Kula   Rasmus WĂŠrn https://www.architectural-review.com/awards/ar-house/house-on-hamburgo-in-bohuslan-sweden-by-manthey-kula
5   El Garaje in Madrid, Spain by Nomos Miguel FernĂĄndez-Galiano    https://www.architectural-review.com/awards/ar-house/el-garaje-in-madrid-spain-by-nomos
6   Water Tower in Castle Acre, UK by Tonkin Liu    Nile Bridgeman  https://www.architectural-review.com/awards/new-into-old/water-tower-in-castle-acre-uk-by-tonkin-liu
7   BAAN 38 in Bangkok, Thailand by Junsekino Architect and Design  Madeleine Weavers   https://www.architectural-review.com/awards/new-into-old/baan-38-in-bangkok-thailand-by-junsekino-architect-and-design
8   Yamato House in Nara, Japan by Tadashi Yoshimura Architects Nancy (Yao) Ji  https://www.architectural-review.com/places/japan/yamato-house-in-nara-japan-by-tadashi-yoshimura-architects
9   Baltrasna House in Dublin, Ireland by Ryan W Kennihan Architects    Lisa Godson https://www.architectural-review.com/places/ireland/baltrasna-house-in-dublin-ireland-by-ryan-w-kennihan-architects
10  Couldrey House in Brisbane, Australia by HNNA and Peter Besley  Kirsty Volz https://www.architectural-review.com/awards/ar-house/couldrey-house-in-brisbane-australia-by-hnna-and-peter-besley
11  Big Space, Little Space in Buffalo, United States by Davidson Rafailidis    Raymund Ryan    https://www.architectural-review.com/awards/ar-house/big-space-little-space-in-buffalo-united-states-by-davidson-rafailidis
12  Home grown: Arigho Larmour Wheeler Architects, Ireland  Marwa El Mubark https://www.architectural-review.com/awards/ar-emerging/home-grown-arigho-larmour-wheeler-architects-ireland
13  Polished conversation: Clancy Moore, Ireland    Lili Zarzycki   https://www.architectural-review.com/awards/ar-emerging/polished-conversation-clancy-moore-ireland
14  Solo House, Matarraña, Spain, by Pezo von Ellrichshausen    AR Editors  https://www.architectural-review.com/buildings/solo-house-matarrana-spain-by-pezo-von-ellrichshausen
15  Solo House, Matarraña, Spain, by OFFICE KGDVS   AR Editors  https://www.architectural-review.com/buildings/solo-house-matarrana-spain-by-office-kgdvs
16  Holiday pay: vacation homes by Living Architecture in the U.K. and Solo Houses in Matarraña, Spain  Manon Mollard   https://www.architectural-review.com/buildings/holiday-pay-vacation-homes-by-living-architecture-in-the-u-k-and-solo-houses-in-matarrana-spain
17  Shadow play: Hawthorn House, Victoria, Australia, by Edition Office Samson SĂ©yĂ­ Famusan https://www.architectural-review.com/awards/ar-house/shadow-play-hawthorn-house-victoria-australia-by-edition-office
18  Balancing blocks: Casa St Cugat, Catalonia, Spain, Camps Felip Arquitecturia    Siufan Adey https://www.architectural-review.com/awards/ar-house/balancing-blocks-casa-st-cugat-catalonia-spain-camps-felip-arquitecturia
19  Spotlight: house in a garden, London, UK, Gianni Botsford Architects    Nile Bridgeman  https://www.architectural-review.com/awards/ar-house/spotlight-house-in-a-garden-london-uk-gianni-botsford-architects
20  Light room: photographer’s house, London, UK, Hugh Strange Architects   Shukri Sultan   https://www.architectural-review.com/awards/ar-house/light-room-photographers-house-london-uk-hugh-strange-architects
21  Ascending order: Loop Terrace, Kobe, Japan, Tomohiro Hata Architect and Associates  Thomas Aquilina https://www.architectural-review.com/awards/ar-house/ascending-order-loop-terrace-kobe-japan-tomohiro-hata-architect-and-associates
22  Holding the line: The Longhouse, Daylesford, Australia, by Partners Hill    Josh Fenton https://www.architectural-review.com/awards/ar-house/holding-the-line-the-longhouse-daylesford-australia-by-partners-hill
23  Slice of life: the Cut, Koganecho, Japan, by Persimmon Hills Architects Aoi Phillips    https://www.architectural-review.com/awards/ar-house/slice-of-life-the-cut-koganecho-japan-by-persimmon-hills-architects
24  Brick house: Casa Volta, Oaxaca, Mexico by Ambrosi Etchegaray   Marwa El Mubark https://www.architectural-review.com/awards/ar-house/brick-house-casa-volta-oaxaca-mexico-by-ambrosi-etchegaray
25  Fringe benefits: Song house, Shanghai, China, by AZL Architects Harry den Hartog    https://www.architectural-review.com/awards/ar-house/fringe-benefits-song-house-shanghai-china-by-azl-architects
26  Stepping stone: Hamra Studio residence, Gotland, Sweden, by Collectif Encore    Lili Zarzycki   https://www.architectural-review.com/awards/ar-house/stepping-stone-hamra-studio-residence-gotland-sweden-by-collectif-encore
27  Cut a dash: Hollybrook Grove in Dublin, Ireland, by David Leech Architects  Eleanor Beaumont    https://www.architectural-review.com/awards/ar-house/cut-a-dash-hollybrook-grove-in-dublin-ireland-by-david-leech-architects
28  Noreile Breen: ‘I am from this place, I have a responsibility to it’    Rupert Bickersteth  https://www.architectural-review.com/archive/noreile-breen-i-am-from-this-place-i-have-a-responsibility-to-it
29  Footprints in time: House in Kamitomii, Kurashiki, Japan, by General Design Co  Kaz Yoneda  https://www.architectural-review.com/awards/ar-house/footprints-in-time-house-in-kamitomii-kurashiki-japan-by-general-design-co
30  Domestic bliss: houses in Dublin by Clancy Moore, TAKA, Ryan W Kennihan Architects and Arigho Larmour Wheeler Architects    Eleanor Beaumont    https://www.architectural-review.com/buildings/domestic-bliss-houses-in-dublin-by-clancy-moore-taka-ryan-w-kennihan-architects-and-arigho-larmour-wheeler-architects
31  Travel agency: Irene PĂ©rez, TEd'A Arquitectes, Spain    Lili Zarzycki   https://www.architectural-review.com/buildings/travel-agency-irene-perez-teda-arquitectes-spain
32  Impractical genius: JĂžrn Utzon's Sydney Opera House and Can Lis, Mallorca   Gerard Reinmuth https://www.architectural-review.com/buildings/impractical-genius-jorn-utzons-sydney-opera-house-and-can-lis-mallorca
33  Nature nurtured: Carla Juaçaba Studio, Brazil   Manon Mollard   https://www.architectural-review.com/today/nature-nurtured-carla-juacaba-studio-brazil
34  Front of house: T O B Architect, Ireland    Eleanor Beaumont    https://www.architectural-review.com/buildings/front-of-house-t-o-b-architect-ireland
35  HĂŽtel Wolfers: Henry van de Velde's frozen ruin Eleanor Beaumont    https://www.architectural-review.com/buildings/hotel-wolfers-henry-van-de-veldes-frozen-ruin
36  Shadow box: House V-D, Ghent, by Marie-José Van Hee Architecten Helen Thomas    https://www.architectural-review.com/buildings/shadow-box-house-v-d-ghent-by-marie-jose-van-hee-architecten
37  Inside the Box: Kokoon by Helsinki’s Aalto University Wood Program Students, Finland    George Kafka    https://www.architectural-review.com/awards/ar-house/inside-the-box-kokoon-by-helsinkis-aalto-university-wood-program-students-finland
38  Pioneer Spirit: Casa Bruma by Fernanda Canales and Claudia RodrĂ­guez in Mexico  Lili Carr   https://www.architectural-review.com/awards/ar-house/pioneer-spirit-casa-bruma-by-fernanda-canales-and-claudia-rodriguez-in-mexico
39  Overarching Craft: Mesura's house extension in Elche, Spain Rafael GĂłmez-Moriana    https://www.architectural-review.com/awards/ar-house/overarching-craft-mesuras-house-extension-in-elche-spain
40  Planes and plants: Vo Trong Nghia Architects’ Binh House in Ho Chi Minh City    Josh Comaroff   https://www.architectural-review.com/awards/ar-house/planes-and-plants-vo-trong-nghia-architects-binh-house-in-ho-chi-minh-city
41  Building in potential: house in Nobeoka, Japan by Schemata Architects   Jan Vranovsk√Äč  https://www.architectural-review.com/awards/ar-house/building-in-potential-house-in-nobeoka-japan-by-schemata-architects
42  Veiled in Secrecy: The Habitat for Orphan Girls by ZAV Architects, Iran Mehr Shafiei    https://www.architectural-review.com/awards/ar-house/veiled-in-secrecy-the-habitat-for-orphan-girls-by-zav-architects-iran
43  Overarching ambition: Two houses, Oropesa, Spain by Paredes Pedrosa Arquitectos Andrew Ayers    https://www.architectural-review.com/buildings/overarching-ambition-two-houses-oropesa-spain-by-paredes-pedrosa-arquitectos
44  Winds of change: Casa de Los Vientos in Cádiz, Spain by José Luis Muñoz  Manon Mollard   https://www.architectural-review.com/awards/ar-emerging/winds-of-change-casa-de-los-vientos-in-cadiz-spain-by-jose-luis-mun%cc%83oz
45  Economy class: Mrs Fan’s Plugin House in Beijing, China by People’s Architecture Office Manon Mollard   https://www.architectural-review.com/awards/ar-emerging/economy-class-mrs-fans-plugin-house-in-beijing-china-by-peoples-architecture-office
46  Sense of an ending: Red House in London by 31/44 Architects Eleanor Beaumont    https://www.architectural-review.com/awards/ar-emerging/sense-of-an-ending-red-house-in-london-by-31-44-architects
47  Dachaland: the Russian dacha    DK-CM   https://www.architectural-review.com/buildings/dachaland-the-russian-dacha
48  Iturbide Studio in Mexico City by Taller Mauricio Rocha + Gabriela Carrillo Michael Webb    https://www.architectural-review.com/buildings/iturbide-studio-in-mexico-city-by-taller-mauricio-rocha-gabriela-carrillo
49  A host of oasts: Caring Wood by James Macdonald Wright with Niall Maxwell   Jay Merrick https://www.architectural-review.com/buildings/a-host-of-oasts-caring-wood-by-james-macdonald-wright-with-niall-maxwell
50  Coogee House, Sydney, Australia by Chenchow Little Architects   Eleanor Beaumont    https://www.architectural-review.com/awards/ar-house/coogee-house-sydney-australia-by-chenchow-little-architects
51  The Captain's House, Fuzhou, China by Vector Architects Eleanor Beaumont    https://www.architectural-review.com/awards/ar-house/the-captains-house-fuzhou-china-by-vector-architects

That 50 limit is arbitrary, you can go higher and get more properties.
See relevant documentation for pandas: https://pandas.pydata.org/docs/

Also for Requests: https://requests.readthedocs.io/en/latest/

And for tqdm, visit https://pypi.org/project/tqdm/

Answered By: Barry the Platipus