Python Selenium – Select Options not returning all the options

Question

I’ve been trying to use Python and Selenium to scrape some prices off a certain website
The script I wrote seems to work well but sometimes I get an error on this part of the code :

    #Select "Date de début"
    select = Select(driver.find_element("xpath",'//*[@id="startDateSet"]'))
    options = select.options
    option = options[i]
    select.select_by_visible_text(option.text)
    print (option.text)
    ListDateEfPrincipal3Semaines.append(option.text)

Error

I don’t get why this happens sometimes, normally it should just get the options list (which contains at least 10 elements and then extract those infos but here it seems the returned list contains just 1 element.

here’s the mentionned Select class :
Select Class Image

here’s the full code :

print ("nDébut du Programme EF Principal pour 3 semaines !")
ListPriceEfPrincipal3Semaines = []
ListDateEfPrincipal3Semaines = []
for i in range(1, 10):
    print("n")
    #add options to the Chrome tab
    options = webdriver.ChromeOptions()
    options.add_argument("--start-maximized")
    driver = webdriver.Chrome(executable_path=path, options=options)
    driver.implicitly_wait(10) #wait max 5 seconds for the cookie page to appear, if it finds it sooner it continues
    
    driver.get('https://www.ef.fr/ils/book-now/')
    driver.find_element("xpath",'//*[@id="onetrust-accept-btn-handler"]').click()

    #Select "Destination"
    select = Select(driver.find_element("xpath",'//*[@id="destination"]'))
    select.select_by_visible_text('Tokyo')

    #Select "Formule de cours"
    select = Select(driver.find_element("xpath",'//*[@id="courseSet"]'))
    select.select_by_visible_text('Programme EF Principal')

    #Select "Durée du séjour"
    select = Select(driver.find_element("xpath",'//*[@id="durationSet"]'))
    select.select_by_visible_text('3 Semaines')

    #Select "Date de début"
    select = Select(driver.find_element("xpath",'//*[@id="startDateSet"]'))
    options = select.options
    option = options[i]
    select.select_by_visible_text(option.text)
    print (option.text)
    ListDateEfPrincipal3Semaines.append(option.text)

    #Click on "Suivant"
    driver.find_element("xpath",'//*[@id="btn-next"]').click()

    #Pick the final price
    content = driver.find_element("class name",'Summary_priceInfo__Wadbp')
    #content = driver.find_element("xpath",'//div[@class="Summary_priceInfo__Wadbp"]')
    FinalPrice = content.text
    print("Prix final : ")
    print(FinalPrice)
    ListPriceEfPrincipal3Semaines.append(FinalPrice)
    driver.close()

print ("nFin du Programme EF Principal pour 3 semaines !n")


print ("nDébut du Programme EF estival pour 3 semaines !")
ListPriceEfEstival3Semaines = []
ListDateEfEstival3Semaines = []
for i in range(1, 10):
    print("n")
    #add options to the Chrome tab
    options = webdriver.ChromeOptions()
    options.add_argument("--start-maximized")
    driver = webdriver.Chrome(executable_path=path, options=options)
    driver.implicitly_wait(10) #wait max 5 seconds for the cookie page to appear, if it finds it sooner it continues
    
    driver.get('https://www.ef.fr/ils/book-now/')
    driver.find_element("xpath",'//*[@id="onetrust-accept-btn-handler"]').click()

    #Select "Destination"
    select = Select(driver.find_element("xpath",'//*[@id="destination"]'))
    select.select_by_visible_text('Tokyo')

    #Select "Formule de cours"
    select = Select(driver.find_element("xpath",'//*[@id="courseSet"]'))
    select.select_by_visible_text('Programme EF estival')

    #Select "Durée du séjour"
    select = Select(driver.find_element("xpath",'//*[@id="durationSet"]'))
    select.select_by_visible_text('3 Semaines')

    #Select "Date de début"
    select = Select(driver.find_element("xpath",'//*[@id="startDateSet"]'))
    options = select.options
    option = options[i]
    select.select_by_visible_text(option.text)
    print (option.text)
    ListDateEfEstival3Semaines.append(option.text)

    #Click on "Suivant"
    driver.find_element("xpath",'//*[@id="btn-next"]').click()

    #Pick the final price
    content = driver.find_element("class name",'Summary_priceInfo__Wadbp')
    #content = driver.find_element("xpath",'//div[@class="Summary_priceInfo__Wadbp"]')
    FinalPrice = content.text
    print("Prix final : ")
    print(FinalPrice)
    ListPriceEfEstival3Semaines.append(FinalPrice)
    driver.close()


print ("nFin du Programme EF estival pour 3 semaines !n")

and here’s the output I get :

runfile('D:/EFPrice_Automate.py', wdir='D:')

Début du Programme EF Principal pour 3 semaines !


d:efprice_automate.py:15: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  driver = webdriver.Chrome(executable_path=path, options=options)
Traceback (most recent call last):

  File "D:anaconda3libsite-packagesspyder_kernelspy3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "d:efprice_automate.py", line 36, in <module>
    option = options[i]

IndexError: list index out of range

Asked By: Mamask Gold

||

Source

Answer 1

The issue might be due to the page not fully loading before you are trying to select an option from the drop-down menu. To fix this, you can try using an explicit wait using the WebDriverWait it will wait until the drop-down options are available before trying to select one. Here is an example:

# Wait until the drop-down options are available
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

select = Select(driver.find_element("xpath",'//*[@id="startDateSet"]'))

wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located(("xpath",'//*[@id="startDateSet"]/option')))

options = select.options
option = options[i]
select.select_by_visible_text(option.text)
print(option.text)
ListDateEfPrincipal3Semaines.append(option.text)

This will wait for up to 10 seconds for the drop-down options to become available. If the options don’t become available within that time, a TimeoutException will be raised.

Answered By: Andrey Kiselev

Python Selenium – Select Options not returning all the options

Question:

Answers: