xpath give empty output using selenium

Question

I am not getting price they give me empty output this is page link https://www.amazon.com/dp/B00M0DWQYI?th=1

from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support.select import Select
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd


url='https://www.amazon.com/dp/B00M0DWQYI?th=1'
PATH="C:Program Files (x86)chromedriver.exe"
driver =webdriver.Chrome(PATH)
driver.get(url)
item=dict()
try:
    item['price'] = driver.find_element(By.XPATH, "//div[@id='corePrice_feature_div'] //span[@class='a-offscreen']").text
except:
    item['price']=''
    
print(item)

Asked By: developer

||

Source

Answer 1

You may want to wait for that element to properly load, prior to locating it:

[...]
wait = WebDriverWait(driver, 10)


item['price'] = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[@id='corePrice_feature_div']//span[@class='a-offscreen']"))).text

Selenium documentation can be found at https://www.selenium.dev/documentation/

EDIT: Here is a complete example of how you can get that information:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
import time as t 

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1920,1080")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
driver = webdriver.Chrome(service=webdriver_service, options=chrome_options)
wait = WebDriverWait(driver, 5)

items = dict()

driver.get('https://www.amazon.com/dp/B00M0DWQYI?th=1')
t.sleep(1)
driver.refresh()
items['price'] = wait.until(EC.element_to_be_clickable((By.XPATH, '//div[@id="corePrice_feature_div"]//span[@class="a-price aok-align-center"]'))).text.replace('n', '.')
print(items)

Result in terminal:

{'price': '$32.98'}

Answered By: Barry the Platipus

Answer 2

You need to wait for element visibility and then to extract it’s text.
The following Selenium code works:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:webdriverschromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = 'https://www.amazon.com/dp/B00M0DWQYI'
driver.get(url)
wait = WebDriverWait(driver, 10)

print(wait.until(EC.visibility_of_element_located((By.XPATH, "//div[@id='corePrice_feature_div']"))).text)

The output is

$32
98

Answered By: Prophet

Answer 3

You can use bs4 and it will work fine

from bs4 import BeautifulSoup
soup = BeautifulSoup(driver.page_source, 'lxml')

try:
    item['price'] = soup.find('input', id="attach-base-product-price").get('value')
except:
   item['price'] = ''
   finally:
           driver.close()
           driver.quit()

print(item)

Answered By: Nero

xpath give empty output using selenium

Question:

Answers: