How to click a label element utilizing Python and Selenium

Question:

I have been building a webscrapper bot with Python and Selenium, but have run into an issue. The website I am using to scrape information has a fieldset html tag with 4 label tags. I am specifically looking to click one of these label tags, but all of them have the same class name. I was hoping to see if anyone had some insight on how I can work around this issue. I have tried out a couple of things but unfortunately none are working.

This is the HTML code:

<fieldset>
                            <legend class="sr-only">Are you a?</legend>
                            <input type="hidden" name="_token" value="Nw3CU75ZNiiwSvcXofVUm3hOpcFkvzEc8UdjlPcX">
                            <span id="errMsg"></span>
                            <div aria-live="assertive" class="error-message"></div>
                            <label class="custom-control custom-radio" tabindex="0" for="radio1">
                                <input tabindex="-1" id="radio1" name="radioStep1" type="radio" value='in-network'
                                    class="custom-control-input" aria-describedby="in-network-balloon">
                                <span class="fa-stack" style="vertical-align: top;">
                                    <i class="fal fa-circle fa-stack-2x"></i>
                                    <i style="color:#fff" class="fa fa-circle fa-stack-1x"></i>
                                </span>
                                <span class="custom-control-description" id='in-network-balloon'>
                                    Staying In-Network <span class='sr-only'>If your provider takes your insurance</span>
                                    <i class="far fa-question-circle float-right" data-toggle="tooltip"
                                        data-html="true" data-trigger="focus keypress click" aria-hidden="true"
                                        title="<h3>Staying In-Network</h3>
                                        <p>If your provider takes your insurance</p>"></i>
                                </span>
                                <mark class="inline-cost-div arrow-btn">
                                    <span class="pt-2 border-top border-white">Next
                                        <i class="fa fa-chevron-square-right"></i>
                                    </span>
                                </mark>
                            </label>
                            <label class="custom-control custom-radio" tabindex="0" for="radio2">
                                <input tabindex="-1" id="radio2" name="radioStep1" type="radio" value='out-network'
                                    class="custom-control-input" aria-describedby="out-network-balloon">
                                <span class="fa-stack" style="vertical-align: top;">
                                    <i class="fal fa-circle fa-stack-2x"></i>
                                    <i style="color:#fff" class="fa fa-circle fa-stack-1x"></i>
                                </span>
                                <span class="custom-control-description" id='out-network-balloon'>
                                    Going Out-of-Network <span class='sr-only'>If your provider does not take your insurance.</span>
                                    <i class="far fa-question-circle float-right" data-toggle="tooltip"
                                        data-html="true" data-trigger="focus keypress click" aria-hidden="true"
                                        title="<h3>Going Out-of-Network</h3>
                                        <p>If your provider does not take your insurance.</p>"></i>
                                </span>
                                <mark class="inline-cost-div arrow-btn">
                                    <span class="pt-2 border-top border-white">Next
                                        <i class="fa fa-chevron-square-right"></i>
                                    </span>
                                </mark>
                            </label>
                            <label class="custom-control custom-radio selected" tabindex="0" for="radio3">
                                <input checked tabindex="-1" id="radio3" name="radioStep1" type="radio" value='not-sure'
                                    class="custom-control-input" aria-describedby="not-sure-balloon">
                                <span class="fa-stack" style="vertical-align: top;">
                                    <i class="fal fa-circle fa-stack-2x"></i>
                                    <i style="color:#fff" class="fa fa-circle fa-stack-1x"></i>
                                </span>
                                <span class="custom-control-description" id="not-sure-balloon">
                                    Not Sure <span class='sr-only'>If you don't know if your provider takes your insurance</span>
                                    <i class="far fa-question-circle float-right" data-toggle="tooltip"
                                        data-html="true" data-trigger="focus keypress click" aria-hidden="true"
                                        title="<h3>Not Sure</h3>
                                        <p>If you don't know if your provider takes your insurance</p>"></i>
                                </span>
                                <mark class="inline-cost-div arrow-btn">
                                    <span class="pt-2 border-top border-white">
                                        Next
                                        <i class="fa fa-chevron-square-right"></i>
                                    </span>
                                </mark>
                            </label>
                            <label class="custom-control custom-radio" tabindex="0" for="radio4">
                                <input tabindex="-1" id="radio4" name="radioStep1" type="radio" value='uninsured'
                                    class="custom-control-input" aria-describedby="uninsured-balloon">
                                <span class="fa-stack" style="vertical-align: top;">
                                    <i class="fal fa-circle fa-stack-2x"></i>
                                    <i style="color:#fff" class="fa fa-circle fa-stack-1x"></i>
                                </span>
                                <span class="custom-control-description" id="uninsured-balloon">
                                    Uninsured <span class="sr-only">If you don't have health insurance or if your insurance pays for providers who aren’t in your plan.</span>
                                    <i class="far fa-question-circle float-right" data-toggle="tooltip"
                                        data-html="true" data-trigger="focus keypress click" aria-hidden="true"
                                        title="<h3>Uninsured</h3>
                                        <p>If you don't have health insurance or if your insurance pays for providers who aren’t in your plan.</p>"></i>
                                </span>
                                <mark class="inline-cost-div arrow-btn">
                                    <span class="pt-2 border-top border-white">
                                        Next
                                        <i class="fa fa-chevron-square-right"></i>
                                    </span>
                                </mark>
                            </label>
                        </fieldset>

Here is an image of what the page looks like. I am specifically looking to press the "Not Sure" section, and then "next"

enter image description here

I have tried this code

    label_elements = driver.find_elements_by_css_selector("custom-control custom-radio")
    label_elements[1].click()

along with this

label_elements = driver.find_elements_by_css_selector("custom-control custom-radio")

for label_element in label_elements:
    label_element.click()

and a couple different variations of the above code, but it still has not been working.

Here is my full code:

import selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
import time
import os
from selenium.webdriver.common.action_chains import ActionChains

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

s=Service('/Users/[name]/Desktop/chromedriver')

chromeOptions = Options()
chromeOptions.headless = False
driver = webdriver.Chrome(service=s, options=chromeOptions)


list_data = []

def initalize_browser():
    driver.get("https://www.fairhealthconsumer.org/")
    driver.maximize_window()
   
    driver.find_element(By.XPATH, '//button[@class="mx-auto green-btn btnHref"]').click()
    
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@id='radio1' and normalize-space()='Einloggen']"))).click()
    driver.find_element(By.ID, 'not-sure-balloon').click()
    driver.find_elements(By.CLASS_NAME, 'fa-chevron-square-right')[2].click()

    print(driver.page_source)

initalize_browser()


driver.quit

Here is the link: fairhealthconsumer.org/medical

Any guidance is super appreciated, thank you!

Answers:

why not use this for the ‘not sure’ button:

driver.find_element_by_id('not-sure-balloon').click()

and then this for the ‘next’ button

driver.find_elements_by_class_name('fa-chevron-square-right')[2].click()

Though this is usually the recommended way:

from selenium.webdriver.common.by import By
...
driver.find_element(By.ID, 'not-sure-balloon').click()
driver.find_elements(By.CLASS_NAME, 'fa-chevron-square-right')[2].click()
Answered By: Andrew Ryan

Try this:

list_data = []
    
def initalize_browser():
    driver.get("https://www.fairhealthconsumer.org/")
    driver.maximize_window()

    driver.find_element(By.XPATH, '//button[@class="mx-auto green-btn btnHref"]').click()

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, ".//*[@for='radio3']/span"))).click()
    driver.find_element(By.XPATH, ".//*[@for='radio3']//span[@class='pt-2 border-top border-white']").click()
    print(driver.page_source)

initalize_browser()
Answered By: AbiSaran

The id of the not sure button is: not-sure-balloon

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service


s = Service("D:chromedriver_win32chromedriver.exe")
browser = webdriver.Chrome(service=s)
url = "https://fairhealthconsumer.org/medical"

browser.get(url)


browser.find_element(By.ID, 'not-sure-balloon').click()
Answered By: Leo Ward
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.