Accepting cookies when web scraping
Question:
Related to a previous question I am trying to edit the answer to apply to another website, but can’t get it to work. What I want to do here is to accept the cookie, and then extract the information from the table. (I also want to scrape the table for all of 2021 later, so any tips on how to proceed there is welcomed too).
from selenium import webdriver
import time
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_experimental_option("detach", True)#optional
webdriver_service = Service("./chromedriver") #Your chromedriver path
driver = webdriver.Chrome(service=webdriver_service,options=options)
data = []
driver.get('https://www.nordpoolgroup.com/en/Market-data1/Power-system-data/Consumption1/Consumption-prognosis/SE/Hourly/?view=table')
time.sleep(3)
cookie = tbutton = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@class="pure-button"]'))).click()
driver.execute_script("arguments[0].click();", tbutton)
time.sleep(1)
soup = BeautifulSoup(driver.page_source,"html.parser")
df = pd.read_html(str(soup))[0]
print(df)
JavascriptException: javascript error: Cannot read properties of null (reading ‘click’)
(Session info: chrome=107.0.5304.107)
I inspected the "I accept cookies" button and it seems that "pure-button" should be inserted in the class field. What could be the issue here?
Thank you.
Answers:
The click() method returns null, so this expression WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@class="pure-button"]'))).click() returns null, so cookie and tbutton are null objects.
Then you trying to click a null object with driver.execute_script("arguments[0].click();", tbutton) and this line gives you an error.
So, in your code you should remove cookie = tbutton = and driver.execute_script("arguments[0].click();", tbutton) while this line
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@class="pure-button"]'))).click()
is enough. It should close the cookies.
That’s it.
Also, this line can be improved.
Since you are clicking that button it’s better to use element_to_be_clickable expected condition than visibility_of_element_located. So, I’d advice this line to be:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@class="pure-button"]'))).click()
Related to a previous question I am trying to edit the answer to apply to another website, but can’t get it to work. What I want to do here is to accept the cookie, and then extract the information from the table. (I also want to scrape the table for all of 2021 later, so any tips on how to proceed there is welcomed too).
from selenium import webdriver
import time
from bs4 import BeautifulSoup
import pandas as pd
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_experimental_option("detach", True)#optional
webdriver_service = Service("./chromedriver") #Your chromedriver path
driver = webdriver.Chrome(service=webdriver_service,options=options)
data = []
driver.get('https://www.nordpoolgroup.com/en/Market-data1/Power-system-data/Consumption1/Consumption-prognosis/SE/Hourly/?view=table')
time.sleep(3)
cookie = tbutton = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@class="pure-button"]'))).click()
driver.execute_script("arguments[0].click();", tbutton)
time.sleep(1)
soup = BeautifulSoup(driver.page_source,"html.parser")
df = pd.read_html(str(soup))[0]
print(df)
JavascriptException: javascript error: Cannot read properties of null (reading ‘click’)
(Session info: chrome=107.0.5304.107)
I inspected the "I accept cookies" button and it seems that "pure-button" should be inserted in the class field. What could be the issue here?
Thank you.
The click() method returns null, so this expression WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@class="pure-button"]'))).click() returns null, so cookie and tbutton are null objects.
Then you trying to click a null object with driver.execute_script("arguments[0].click();", tbutton) and this line gives you an error.
So, in your code you should remove cookie = tbutton = and driver.execute_script("arguments[0].click();", tbutton) while this line
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//*[@class="pure-button"]'))).click()
is enough. It should close the cookies.
That’s it.
Also, this line can be improved.
Since you are clicking that button it’s better to use element_to_be_clickable expected condition than visibility_of_element_located. So, I’d advice this line to be:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[@class="pure-button"]'))).click()