Why can't I retrieve the price from a webpage?

Question

I am trying to retrieve a price from a product on a webshop but can’t find the right code to get it.

Price of product I want to extract: https://www.berger-camping.nl/zoeken/?q=3138522088064

This is the line of code I have to retrieve the price:

Prijs_BergerCamping = driver.find_element(by=By.XPATH, value='//div[@class="prod_price__prod_price"]').text
        print(Prijs_BergerCamping)

Any tips on what I seem to be missing?

Asked By: Dion

||

Source

Answer 1

Your code is correct.
I guess all you missing is to wait for element visibility.
This code works:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:webdriverschromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)

url = "https://www.berger-camping.nl/zoeken/?q=3138522088064"
driver.get(url)

price = wait.until(EC.visibility_of_element_located((By.XPATH, '//div[@class="prod_price__prod_price"]'))).text
print(price)

The output is:

79,99 €

But you also need to close the cookies banner and Select store dialog (at least I see it). So, my code has the following additionals:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:webdriverschromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 20)

url = "https://www.berger-camping.nl/zoeken/?q=3138522088064"
driver.get(url)

wait.until(EC.element_to_be_clickable((By.ID, "onetrust-accept-btn-handler"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@style] //*[contains(@class,'uk-close')]"))).click()
price = wait.until(EC.visibility_of_element_located((By.XPATH, '//div[@class="prod_price__prod_price"]'))).text
print(price)

Answered By: Prophet

Why can't I retrieve the price from a webpage?

Question:

Answers: