Python selenium how to wait until element is gone/becomes stale?
Question:
I am trying to programmatically solve the problem presented at this address: https://www.arealme.com/brain-memory-game/en/, in short it will display several flashing numbers in a row, and asking you to input the numbers in reverse order.
Now here are the details, when you get to the website, there will be a start button which has a blue bar that fills it from the left, I have found that if you let your bot click that button before the blue bar fills up the button, then the function of the button won’t be triggered and the rest of the script won’t run.
After the start button is clicked, the page will refresh, and you will see this flashing text:
The numbers are coming soon. Please pay attention…
And after that, you will see several flashing numbers, they flash by changing alpha, they change from fully visible to fully invisible, when a number disappears another number will appear, they are different numbers even if they have the same value.
Then there will be this message:
Please click on the number you just saw IN REVERSE ORDER.
And a number pad with ten buttons.
Do as it said and then a button will appear that tells whether you are right or not and you need to click it to proceed to the next question.
There are ten questions in total.
Now with the technical details.
I have managed to find the class names and xpaths and ids of the elements involved.
The id of the start button is 'start', and I have found time.sleep(3) is sufficient for it to fill up.
The hierarchy of the classes is as follows:
<div class="questionWrapper">
<div class="question">
<div class="gnumber_title">
<div class="gend-tip">Please click on the number you just saw IN REVERSE ORDER.</div>
</div>
<div class="gnumber_btns" id="gbtn0" style=""><button data-n="9">9</button><button data-n="8">8</button><button data-n="7">7</button><button data-n="6">6</button><button data-n="5">5</button><button data-n="4">4</button><button data-n="3">3</button><button data-n="2">2</button><button data-n="1">1</button><button data-n="0">0</button></div>
</div>
<div class="answer" value="10">1</div>
</div>
The questions are wrapped in "questionWrapper" class, in each there is one and only one instance of class "gnumber_title", the object contains the messages and numbers.
At all times the class holds exactly one element, what the class holds changes with time, but they all share the same xpath relative to said class: './div'
At the start of questions, when the prompt is the first message, the object inside the said location is of class "gstart-tip blink_me".
Then it will disappear and in its place there will be an object of class 'gflash-num', whose content can be accessed using .text attribute.
It blinks by changing style, the style will change from "opacity: 1.000000;" to "opacity: 0.000000;" (I don’t know the exact values, but it is a float with six decimal places from 1 to 0), then it will become "display: none;", then it will be deleted and another instance of "glash-num" will appear.
After several instances of "gflash-num", the second message appears, and its class is "gend-tip", located at the same xpath as the numbers and first message.
And the number pad will be visible, whose class is "gnumber_btns".
After the same number of displayed numbers of the buttons have been clicked, the button whose class is "answer" appears, clicking it will proceed to the next question.
Here is my attempt to solve the problem:
import time
from selenium import webdriver
Firefox = webdriver.Firefox()
Firefox.get('https://www.arealme.com/brain-memory-game/en/')
time.sleep(3)
Firefox.find_element_by_id('start').click()
questions = Firefox.find_elements_by_class_name("questionWrapper")
for q in questions:
numbers = []
title = q.find_element_by_class_name('gnumber_title')
while True:
if title.find_element_by_xpath('./div').get_attribute('class') != 'gstart-tip blink_me':
break
while True:
if title.find_element_by_xpath('./div').get_attribute('class') == "gend-tip":
break
numbers.append(title.find_element_by_class_name('gflash-num').text)
while True:
if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('opacity'):
break
while True:
if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('display'):
break
buttons = q.find_element_by_class_name("gnumber_btns")
for n in reversed(numbers):
buttons.find_element_by_xpath(f'.//*[text() = "{n}"]').click()
time.sleep(0.5)
q.find_element_by_class_name("answer").click()
And the errors:
---------------------------------------------------------------------------
StaleElementReferenceException Traceback (most recent call last)
<ipython-input-1-9d34e88c4046> in <module>
20 numbers.append(title.find_element_by_class_name('gflash-num').text)
21 while True:
---> 22 if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('opacity'):
23 break
24 while True:
c:program filespython39libsite-packagesseleniumwebdriverremotewebelement.py in get_attribute(self, name)
137 attributeValue = ''
138 if self._w3c:
--> 139 attributeValue = self.parent.execute_script(
140 "return (%s).apply(null, arguments);" % getAttribute_js,
141 self, name)
c:program filespython39libsite-packagesseleniumwebdriverremotewebdriver.py in execute_script(self, script, *args)
632 command = Command.EXECUTE_SCRIPT
633
--> 634 return self.execute(command, {
635 'script': script,
636 'args': converted_args})['value']
c:program filespython39libsite-packagesseleniumwebdriverremotewebdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
c:program filespython39libsite-packagesseleniumwebdriverremoteerrorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
StaleElementReferenceException: Message: The element reference of <div class="gflash-num"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
The exception can be raised at any iteration of the dynamic element, I have to wait until the element located at that xpath is no longer "gstart-tip blink_me" to start the next stage of code execution, then add the value of "gflash-num" and wait until the element is gone to add the next value, and then finally when it becomes "gend-tip" click the buttons in reverse order.
I have tried to avoid the exception by not assigning variables and getting the attribute on the fly, but the exception still got raised.
But the time when it raises exceptions is exactly when it is supposed to wake up from its sleep and add the numbers.
So how can I wait until an element becomes stale/invisible/deleted/non-existent/whatsoever, all the Google searching tells exactly how to do the opposite: wait until the element is NO LONGER STALE, but I want to wait until the element is GONE, so how to do this?
I think I have found something very important to solving the problem, using ffmpeg to extract frames from screen recording of the process, I am able to determine the exact during of the flashings.
The first prompt shows exactly 3 seconds, and each number flashes exactly 1 second.
Answers:
to wait till any elelemt becomes invisible we have this Expected conditions :
invisibility_of_element
also in code I could see :
class invisibility_of_element(invisibility_of_element_located):
""" An Expectation for checking that an element is either invisible or not
present on the DOM.
element is either a locator (text) or an WebElement
"""
def __init(self, element):
self.target = element
if you have a running webdriverwait object, then you can try this :
WebDriverWait(driver, 10).until(EC.invisibility_of_element((By.XPATH, "xpath here")))
this will wait till the invisibility of element, defined by xpath.
I have solved my problem, without using invisibility_of_element, maybe I circumvented the problem, but the code doesn’t throw exceptions now.
Since I know the exact "lifespans" of the elements, I could use time.sleep() rather than relying on EC.invisibility_of_element() whose behavior might be unpredictable.
The code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Firefox = webdriver.Firefox()
Firefox.get('https://www.arealme.com/brain-memory-game/en/')
time.sleep(3)
Firefox.find_element_by_id('start').click()
questions = Firefox.find_elements_by_class_name("questionWrapper")
wait = WebDriverWait(Firefox, 3)
for i, q in enumerate(questions):
wait.until(EC.visibility_of_element_located((By.ID, f'q{i + 1}')))
numbers = []
ids = set()
title = q.find_element_by_class_name('gnumber_title')
if title.find_element_by_xpath('./div').get_attribute('class') == 'gstart-tip blink_me':
time.sleep(3)
while True:
if title.find_element_by_xpath('./div').get_attribute('class') == "gend-tip":
break
number = title.find_element_by_class_name('gflash-num')
if number.id not in ids:
numbers.append(number.text)
ids.add(number.id)
time.sleep(1)
wait.until(EC.visibility_of_element_located((By.ID, f'gbtn{i}')))
buttons = q.find_element_by_class_name("gnumber_btns")
for n in reversed(numbers):
buttons.find_element_by_xpath(f'.//*[@data-n = "{n}"]').click()
time.sleep(0.1)
time.sleep(0.5)
q.find_element_by_class_name("answer").click()
You can use invisibility_of_element_located() or until_not() methods.
WebDriverWait(self.driver, timeout).until(ec.invisibility_of_element_located(locator))
WebDriverWait(self.driver, timeout).until_not(ec.visibility_of_element_located(locator))
I am trying to programmatically solve the problem presented at this address: https://www.arealme.com/brain-memory-game/en/, in short it will display several flashing numbers in a row, and asking you to input the numbers in reverse order.
Now here are the details, when you get to the website, there will be a start button which has a blue bar that fills it from the left, I have found that if you let your bot click that button before the blue bar fills up the button, then the function of the button won’t be triggered and the rest of the script won’t run.
After the start button is clicked, the page will refresh, and you will see this flashing text:
The numbers are coming soon. Please pay attention…
And after that, you will see several flashing numbers, they flash by changing alpha, they change from fully visible to fully invisible, when a number disappears another number will appear, they are different numbers even if they have the same value.
Then there will be this message:
Please click on the number you just saw IN REVERSE ORDER.
And a number pad with ten buttons.
Do as it said and then a button will appear that tells whether you are right or not and you need to click it to proceed to the next question.
There are ten questions in total.
Now with the technical details.
I have managed to find the class names and xpaths and ids of the elements involved.
The id of the start button is 'start', and I have found time.sleep(3) is sufficient for it to fill up.
The hierarchy of the classes is as follows:
<div class="questionWrapper">
<div class="question">
<div class="gnumber_title">
<div class="gend-tip">Please click on the number you just saw IN REVERSE ORDER.</div>
</div>
<div class="gnumber_btns" id="gbtn0" style=""><button data-n="9">9</button><button data-n="8">8</button><button data-n="7">7</button><button data-n="6">6</button><button data-n="5">5</button><button data-n="4">4</button><button data-n="3">3</button><button data-n="2">2</button><button data-n="1">1</button><button data-n="0">0</button></div>
</div>
<div class="answer" value="10">1</div>
</div>
The questions are wrapped in "questionWrapper" class, in each there is one and only one instance of class "gnumber_title", the object contains the messages and numbers.
At all times the class holds exactly one element, what the class holds changes with time, but they all share the same xpath relative to said class: './div'
At the start of questions, when the prompt is the first message, the object inside the said location is of class "gstart-tip blink_me".
Then it will disappear and in its place there will be an object of class 'gflash-num', whose content can be accessed using .text attribute.
It blinks by changing style, the style will change from "opacity: 1.000000;" to "opacity: 0.000000;" (I don’t know the exact values, but it is a float with six decimal places from 1 to 0), then it will become "display: none;", then it will be deleted and another instance of "glash-num" will appear.
After several instances of "gflash-num", the second message appears, and its class is "gend-tip", located at the same xpath as the numbers and first message.
And the number pad will be visible, whose class is "gnumber_btns".
After the same number of displayed numbers of the buttons have been clicked, the button whose class is "answer" appears, clicking it will proceed to the next question.
Here is my attempt to solve the problem:
import time
from selenium import webdriver
Firefox = webdriver.Firefox()
Firefox.get('https://www.arealme.com/brain-memory-game/en/')
time.sleep(3)
Firefox.find_element_by_id('start').click()
questions = Firefox.find_elements_by_class_name("questionWrapper")
for q in questions:
numbers = []
title = q.find_element_by_class_name('gnumber_title')
while True:
if title.find_element_by_xpath('./div').get_attribute('class') != 'gstart-tip blink_me':
break
while True:
if title.find_element_by_xpath('./div').get_attribute('class') == "gend-tip":
break
numbers.append(title.find_element_by_class_name('gflash-num').text)
while True:
if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('opacity'):
break
while True:
if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('display'):
break
buttons = q.find_element_by_class_name("gnumber_btns")
for n in reversed(numbers):
buttons.find_element_by_xpath(f'.//*[text() = "{n}"]').click()
time.sleep(0.5)
q.find_element_by_class_name("answer").click()
And the errors:
---------------------------------------------------------------------------
StaleElementReferenceException Traceback (most recent call last)
<ipython-input-1-9d34e88c4046> in <module>
20 numbers.append(title.find_element_by_class_name('gflash-num').text)
21 while True:
---> 22 if not title.find_element_by_class_name('gflash-num').get_attribute('style').startswith('opacity'):
23 break
24 while True:
c:program filespython39libsite-packagesseleniumwebdriverremotewebelement.py in get_attribute(self, name)
137 attributeValue = ''
138 if self._w3c:
--> 139 attributeValue = self.parent.execute_script(
140 "return (%s).apply(null, arguments);" % getAttribute_js,
141 self, name)
c:program filespython39libsite-packagesseleniumwebdriverremotewebdriver.py in execute_script(self, script, *args)
632 command = Command.EXECUTE_SCRIPT
633
--> 634 return self.execute(command, {
635 'script': script,
636 'args': converted_args})['value']
c:program filespython39libsite-packagesseleniumwebdriverremotewebdriver.py in execute(self, driver_command, params)
319 response = self.command_executor.execute(driver_command, params)
320 if response:
--> 321 self.error_handler.check_response(response)
322 response['value'] = self._unwrap_value(
323 response.get('value', None))
c:program filespython39libsite-packagesseleniumwebdriverremoteerrorhandler.py in check_response(self, response)
240 alert_text = value['alert'].get('text')
241 raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)
243
244 def _value_or_default(self, obj, key, default):
StaleElementReferenceException: Message: The element reference of <div class="gflash-num"> is stale; either the element is no longer attached to the DOM, it is not in the current frame context, or the document has been refreshed
The exception can be raised at any iteration of the dynamic element, I have to wait until the element located at that xpath is no longer "gstart-tip blink_me" to start the next stage of code execution, then add the value of "gflash-num" and wait until the element is gone to add the next value, and then finally when it becomes "gend-tip" click the buttons in reverse order.
I have tried to avoid the exception by not assigning variables and getting the attribute on the fly, but the exception still got raised.
But the time when it raises exceptions is exactly when it is supposed to wake up from its sleep and add the numbers.
So how can I wait until an element becomes stale/invisible/deleted/non-existent/whatsoever, all the Google searching tells exactly how to do the opposite: wait until the element is NO LONGER STALE, but I want to wait until the element is GONE, so how to do this?
I think I have found something very important to solving the problem, using ffmpeg to extract frames from screen recording of the process, I am able to determine the exact during of the flashings.
The first prompt shows exactly 3 seconds, and each number flashes exactly 1 second.
to wait till any elelemt becomes invisible we have this Expected conditions :
invisibility_of_element
also in code I could see :
class invisibility_of_element(invisibility_of_element_located):
""" An Expectation for checking that an element is either invisible or not
present on the DOM.
element is either a locator (text) or an WebElement
"""
def __init(self, element):
self.target = element
if you have a running webdriverwait object, then you can try this :
WebDriverWait(driver, 10).until(EC.invisibility_of_element((By.XPATH, "xpath here")))
this will wait till the invisibility of element, defined by xpath.
I have solved my problem, without using invisibility_of_element, maybe I circumvented the problem, but the code doesn’t throw exceptions now.
Since I know the exact "lifespans" of the elements, I could use time.sleep() rather than relying on EC.invisibility_of_element() whose behavior might be unpredictable.
The code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Firefox = webdriver.Firefox()
Firefox.get('https://www.arealme.com/brain-memory-game/en/')
time.sleep(3)
Firefox.find_element_by_id('start').click()
questions = Firefox.find_elements_by_class_name("questionWrapper")
wait = WebDriverWait(Firefox, 3)
for i, q in enumerate(questions):
wait.until(EC.visibility_of_element_located((By.ID, f'q{i + 1}')))
numbers = []
ids = set()
title = q.find_element_by_class_name('gnumber_title')
if title.find_element_by_xpath('./div').get_attribute('class') == 'gstart-tip blink_me':
time.sleep(3)
while True:
if title.find_element_by_xpath('./div').get_attribute('class') == "gend-tip":
break
number = title.find_element_by_class_name('gflash-num')
if number.id not in ids:
numbers.append(number.text)
ids.add(number.id)
time.sleep(1)
wait.until(EC.visibility_of_element_located((By.ID, f'gbtn{i}')))
buttons = q.find_element_by_class_name("gnumber_btns")
for n in reversed(numbers):
buttons.find_element_by_xpath(f'.//*[@data-n = "{n}"]').click()
time.sleep(0.1)
time.sleep(0.5)
q.find_element_by_class_name("answer").click()
You can use invisibility_of_element_located() or until_not() methods.
WebDriverWait(self.driver, timeout).until(ec.invisibility_of_element_located(locator))
WebDriverWait(self.driver, timeout).until_not(ec.visibility_of_element_located(locator))