Selenium – unable to locate button on cookie popup

Question

I am trying to parse a website using selenium.
The website is here: https://www.immowelt.at/suche/wohnungen/mieten
I fail to find the buttons of the cookie popup which I need to confirm to proceed. I know that I first need to load the page and then wait for the cookie popup to appear, although that should be well handled by the sleep function.

The html of the buttons inside the popup looks like this (retrieved via firefox F12). I am trying to simply click the "OK" button:

<button role="button" data-testid="uc-customize-button" style="border: 1px solid rgb(0, 139, 2);" class="sc-gsDKAQ hWcdhQ">Einstellungen oder ablehnen</button>
<div class="sc-bBHxTw foBPAO"></div>
<button role="button" data-testid="uc-accept-all-button" style="border: 1px solid rgb(0, 139, 2);" class="sc-gsDKAQ fILFKg">OK</button>

I have tried to simply find the buttons via the xpath of the "OK" button:

from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By

# adapting: https://stackoverflow.com/questions/64032271/handling-accept-cookies-popup-with-selenium-in-python

BASE_URL = "https://www.immowelt.at/suche/wohnungen/mieten"

driver = webdriver.Firefox()
driver.get(BASE_URL)
sleep(10)

# cannot locate by XPATH
driver.find_element(
    By.XPATH, "/div/div/div[2]/div/div[2]/div/div/div/button[2]"
).click()

# weird thing is, I also cannot retrieve any of the buttons (although I know they exist)
buttons = driver.find_elements(By.TAG_NAME, "button")

Any ideas why I can’t find the button?

Asked By: geopanda1

||

Source

Answer 1

By inspecting the HTML we can see that the button is inside a shadow root.

So in order to be able to interact with the button we must first select the parent of the shadow root, which is the div element with id=usercentrics-root, and then we can select the button and click it:

driver.execute_script('''return document.querySelector("#usercentrics-root").shadowRoot.querySelector("button[data-testid='uc-accept-all-button']")''').click()

Alternatively, a more human readable code:

shadow_parent = driver.find_element(By.CSS_SELECTOR, '#usercentrics-root')
outer = driver.execute_script('return arguments[0].shadowRoot', shadow_parent)
outer.find_element(By.CSS_SELECTOR, "button[data-testid='uc-accept-all-button']").click()

Answered By: sound wave

Selenium – unable to locate button on cookie popup

Question:

Answers: