can't select a checkbox in webdriver.Chrome python

Question

I’m trying to create a script to show only pikachus on singapore poke map and the rest of the code is to go over the elements and get the coords for it and print the list.
I’m trying for a long time many suggestions I’ve seen here but still unable to make the checkbox be set with the latest code:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
import time


def find_pokemon():
    links = []
    service = ChromeService(ChromeDriverManager().install())
    driver = webdriver.Chrome(service=service)
    driver.get('https://sgpokemap.com/index.html?fbclid=IwAR2p_93Ll6K9b923VlyfaiTglgeog4uWHOsQksvzQejxo2fkOj4JN_t-MN8')
    driver.find_element(By.ID, 'filter_link').click()
    driver.find_element(By.ID, 'deselect_all_btn').click()
    driver.find_element(By.ID, 'search_pokemon').send_keys("pika")
    driver.switch_to.frame(driver.find_elements(By.ID, "filter"))
    driver.find_element(By.ID, 'checkbox_25').click()

The second part of the code is working when I’m checking the box manually after putting a breakpoint and ignoring the checkbox click() exception.

Do you have any suggestions what can I try?

Bonus question, how can I determine and close the donate view:

Asked By: Aviad

||

Source

Answer 1

There are several problems with your code:

There is no element with ID = ‘search_pokemon’
There is no frame there to switch into it.
You need to use WebDriverWait expected_conditions to wait for elements to be clickable.
And generally you need to learn how to create correct locators.
The following code works:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:webdriverschromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 30)

url = "https://sgpokemap.com/index.html?fbclid=IwAR2p_93Ll6K9b923VlyfaiTglgeog4uWHOsQksvzQejxo2fkOj4JN_t-MN8"
driver.get(url)

try:
    wait.until(EC.element_to_be_clickable((By.ID, 'close_donation_button'))).click()
except:
    pass

wait.until(EC.element_to_be_clickable((By.ID, 'filter_link'))).click()
wait.until(EC.element_to_be_clickable((By.ID, "deselect_all_btn"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[name='search_pokemon']"))).send_keys("pika")
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@class='filter_checkbox'][not(@style)]//label"))).click()

The result is:

UPD

This time I saw the donation dialog so I added the mechanism to close it.
I still can’t see there element with ID = 'search_pokemon' as you mentioned.
As about the XPath to find the relevant checkbox – when pokemon name is inserted you can see in the dev tools that there are a lot of checkboxes there but all of them are invisibly while only one in our case is visible. The invisible elements are all have attribute style="display: none;" while the enabled element does not have style attribute. This is why [not(@style)] is coming there. So, I’m looking for parent element //div[@class='filter_checkbox'] who is also have no style attribute. In XPath words //div[@class='filter_checkbox'][not(@style)] then I’m just looking for it label child to click it. This can also be done with CSS Selectors as well.

The list of invisible elements with the enabled one:

Answered By: Prophet

Answer 2

With the help and answers from @Prophet , the current code for crawling the map and getting all of the Pikachus coordinates:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

from keep import saveToKeep


def find_pokemon():
    links = []
    options = Options()
    options.add_argument("--headless")
    options.add_argument("disable-infobars")
    webdriver_service = Service('C:webdriverschromedriver.exe')
    driver = webdriver.Chrome(options=options, service=webdriver_service)
    wait = WebDriverWait(driver, 30)

    driver.get('https://sgpokemap.com')
    try:
        wait.until(EC.element_to_be_clickable((By.ID, 'close_donation_button'))).click()
    except:
        pass
    wait.until(EC.element_to_be_clickable((By.ID, 'filter_link'))).click()
    wait.until(EC.element_to_be_clickable((By.ID, "deselect_all_btn"))).click()
    wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[name='search_pokemon']"))).send_keys("pika")
    wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@class='filter_checkbox'][not(@style)]//label"))).click()
    # count = 0
    wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'pokemon_icon_img')))
    pokeList = driver.find_elements(By.CLASS_NAME, 'pokemon_icon_img')
    for poke in pokeList:
        # count += 1
        try:
            poke.click()
            links.append(driver.find_element(By.LINK_TEXT, "Maps").get_attribute('href'))
        except Exception:
            pass
        # if count > 300:
        #     break

    res = []
    for link in links:
        res.append(link.split("=")[1].replace("'", ""))

    # for item in res:
    #     print(item)
    if len(res) > 1:
        saveToKeep(res)
        print("success")
    else:
        print("unsuccessful")
        find_pokemon()


if __name__ == '__main__':
    find_pokemon()

Used the headless chrome option in hope to achieve better
performance.
Commented out ‘count’ in case I want to limit list results
(currently I’m getting like 15 results tops when unlimited
although there are many more…weird 🙁 )
the following code wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'pokemon_icon_img'))) is needed for now since it’s not always
showing icons right away, so it’s either that or adding a constant
time delay.
Have made this method recursive in case it’s unsuccessful(sometimes it still gives out exceptions)
Lastly, saveToKeep(res) method is a simple method I’m using to open
and write results into my google keep notes. Needed to get an app
password within google security settings and I’m using it with my google account credentials for login.

Any comments or regards for improvements are welcomed 😀

Answered By: Aviad

can't select a checkbox in webdriver.Chrome python

Question:

Answers: