NoSuchElementException: Message: no such element: Unable to locate element error scraping the Model names using Selenium Python

Question

I’m trying to scrape data from the URL, but when I use the method find_elemnts() I get this error. I am trying to get some data. Here is my code:

from  selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time

#opening the browser
PATH = "C:Program Files (x86)chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://www.bestbuy.com/")

#sending a request through the search bar
search = driver.find_element(By.NAME, 'st')
search.send_keys("dell xps")
search.send_keys(Keys.RETURN)

driver.maximize_window() # For maximizing window
driver.implicitly_wait(20) # gives an implicit wait for 20 seconds

#clicking a desired link by the class name
# driver.find_element(By.CLASS_NAME,'sku-title').click()

# paths:
# sku-title #class name for href links
# //*[@id="main-results"]/ol/li[1]  #link class 
# //*[@id="shop-sku-list-item-23022673"]/div/div/div[1]/div[3]/div[1]/div[1]/span[2] class name: sju-value #model
# //*[@id="pricing-price-23825371"]/div/div/div/div/div[1]/div/div[1]/div class name:priceView-hero-price priceView-customer-price #price

links = driver.find_elements(By.CLASS_NAME, 'sku-title')
for link in links:
    # href = link.find_element(By.XPATH, './/*[@id="main-results"]/ol/li[1]').text
    model = link.find_element(By.CLASS_NAME, 'sku-value').text
    # price = link.find_element(By.XPATH, './/*[@id="pricing-price-23825371"]/div/div/div/div/div[1]/div/div[1]/div').text
    print(model)

time.sleep(200)
driver.quit()

Expecting the data in a text (price, model and the links)

Asked By: omar watad

||

Source

Answer 1

To scrape the Model numbers you need to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:

Code Block:

driver.get("https://www.bestbuy.com/")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.us-link > img"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.search-input"))).send_keys("dell xps")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[title='submit search']"))).click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[@class='attribute-title' and text()='Model:']//following::span[1]")))])
driver.quit()

Console Output:

['XPS9320-7523BLK-PUS', 'XPS9520-9195SLV-PUS', 'XPS9320-7585SLV-PUS', 'XPS9520-7171SLV-PUS', 'XPS9520-7294WHT-PUS', 'BBY-K2PKKFX', 'BBY-46J4FFX', 'XPS9720-7218PLT-PUS', 'BBY-W8PYKFX']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Answered By: undetected Selenium

NoSuchElementException: Message: no such element: Unable to locate element error scraping the Model names using Selenium Python

Question:

Answers: