selenium.common.exceptions.ElementClickInterceptedException error clicking on a toggle button on yahoo finance income statement data
Question:
I am trying to webscrape Income Statement data from https://finance.yahoo.com/quote/AAPL/financials?p=AAPL
I am trying to simulate a click on Operating Expense to expand the row to see the Research and Development values like such:
This is my code so far:
def clickOperatingExpense(url):
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get(url)
# This is where I encounter a selenium.common.exceptions.ElementClickInterceptedException
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="Col1-1-Financials-Proxy"]/section/div[4]/div[1]/div[1]/div[2]/div[4]/div[1]/div[1]/div[1]/button'))).click()
Just for some more information, the full error is:
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <button aria-label="Operating Expense" class="P(0) M(0) Va(m) Bd(0) Fz(s) Mend(2px) tgglBtn">...</button> is not clickable at point (39, 592). Other element would receive the click: <div tabindex="1" class="lightbox-wrapper Ta(c) Pos(f) T(0) Start(0) H(100%) W(100%) Bgc($modalBackground) Ovy(a) Z(50) Op(1)">...</div>
The issue looks like a div tag would receive the click but I’m not sure why. How can I make the tag receive the click instead?
Answers:
Try using Java script to perform click() as below:
driver.execute_script("arguments[0].click();", element)
Reference: https://stackoverflow.com/a/75645682/7598774
To click and expand the element with text as Operating Expense you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
-
Using CSS_SELECTOR:
driver.get("https://finance.yahoo.com/quote/AAPL/financials?p=AAPL")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='Close']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[title='Operating Expense'] > button[aria-label='Operating Expense']"))).click()
-
Using XPATH:
driver.get("https://finance.yahoo.com/quote/AAPL/financials?p=AAPL")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-label='Close']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@title='Operating Expense']/button[@aria-label='Operating Expense']"))).click()
-
Browser Snapshot:
-
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to webscrape Income Statement data from https://finance.yahoo.com/quote/AAPL/financials?p=AAPL
I am trying to simulate a click on Operating Expense to expand the row to see the Research and Development values like such:
This is my code so far:
def clickOperatingExpense(url):
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get(url)
# This is where I encounter a selenium.common.exceptions.ElementClickInterceptedException
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[@id="Col1-1-Financials-Proxy"]/section/div[4]/div[1]/div[1]/div[2]/div[4]/div[1]/div[1]/div[1]/button'))).click()
Just for some more information, the full error is:
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <button aria-label="Operating Expense" class="P(0) M(0) Va(m) Bd(0) Fz(s) Mend(2px) tgglBtn">...</button> is not clickable at point (39, 592). Other element would receive the click: <div tabindex="1" class="lightbox-wrapper Ta(c) Pos(f) T(0) Start(0) H(100%) W(100%) Bgc($modalBackground) Ovy(a) Z(50) Op(1)">...</div>
The issue looks like a div tag would receive the click but I’m not sure why. How can I make the tag receive the click instead?
Try using Java script to perform click() as below:
driver.execute_script("arguments[0].click();", element)
Reference: https://stackoverflow.com/a/75645682/7598774
To click and expand the element with text as Operating Expense you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
-
Using CSS_SELECTOR:
driver.get("https://finance.yahoo.com/quote/AAPL/financials?p=AAPL") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[aria-label='Close']"))).click() WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[title='Operating Expense'] > button[aria-label='Operating Expense']"))).click()
-
Using XPATH:
driver.get("https://finance.yahoo.com/quote/AAPL/financials?p=AAPL") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@aria-label='Close']"))).click() WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@title='Operating Expense']/button[@aria-label='Operating Expense']"))).click()
-
Browser Snapshot:
-
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC