Why is the code not displaying total count of products?

Question

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from time import sleep
from selenium.webdriver.common.by import By


path = r'C:Users.wdmdriverschromedriverwin32108.0.5359chromedriver.exe'

# open the browser
# browser = webdriver.Chrome(executable_path=path)
s=Service(r'C:Users.wdmdriverschromedriverwin32108.0.5359chromedriver.exe')
browser = webdriver.Chrome(service=s)
# load the webpage
browser.get('https://www.amazon.in')
browser.maximize_window()

# get the input elements
input_search = browser.find_element(By.ID,'twotabsearchtextbox')
search_button = browser.find_element(By.XPATH,"(//input[@type='submit'])[1]")

# send the input to the webpage
input_search.send_keys("Smartphones under 10000")
sleep(1)
search_button.click()

products = []
for i in range(10):
    print('Scraping page', i+1)
    product = browser.find_elements(By.CLASS_NAME,'a-size-medium a-color-base a-text-normal')
    for p in product:
        products.append(p.text)
    next_button = browser.find_element(By.CLASS_NAME,'a-last')
    next_button.click()
    sleep(3)

print(len(products))
products[:5]
brower.quit()

Above is the code to scrape the data from pages, and display the total count as well as product names. But this code isn’t displaying the scraped data, what might be the issue?

Here is the Output:

Scraping page 1
Scraping page 2
Scraping page 3
Scraping page 4
Scraping page 5
Scraping page 6
Scraping page 7
Scraping page 8
Scraping page 9
Scraping page 10
0

Process finished with exit code 0

That’s it, no count nothing.

Asked By: Anonymous

||

Source

Answer 1

You have to modify the locators inside the for loop:

# send the input to the webpage
input_search.send_keys("Smartphones under 10000")
search_button.click()
sleep(1)

products = []
for i in range(10):
    # print('Scraping page', i+1)
    product = browser.find_elements(By.CSS_SELECTOR,'.a-size-medium.a-color-base.a-text-normal')
    for p in product:
        products.append(p.text)
    next_button = browser.find_element(By.CSS_SELECTOR,'.s-pagination-item.s-pagination-next')
    next_button.click()
    sleep(3)

print("Total products:", len(products))
for i in range(len(products)):
    print(products[i])

Output:

Total products: 164
Lava Blaze NXT(Glass Red, 4GB RAM, 64GB Storage)| 2.3 Ghz Octa Core Helio G37| 13 MP AI Triple Camera |Fingerprint Sensor| 5000 mAh Battery| Upto 7GB Expandable RAM
Redmi 9 Activ (Carbon Black, 4GB RAM, 64GB Storage) | Octa-core Helio G35 | 5000 mAh Battery
Redmi A1 (Light Blue, 2GB RAM, 32GB Storage) | Segment Best AI Dual Cam | 5000mAh Battery | Leather Texture Design | Android 12
realme narzo 50i (Carbon Black, 2GB RAM+32GB Storage) Octa Core Processor | 6.5" inch Large Display
Redmi 9A Sport (Coral Green, 2GB RAM, 32GB Storage) | 2GHz Octa-core Helio G25 Processor | 5000 mAh Battery
Redmi 10A (Slate Grey, 4GB RAM, 64GB Storage) | 2 Ghz Octa Core Helio G25 | 5000 mAh Battery | Finger Print Sensor | Upto 5GB RAM with RAM Booster
Redmi 9A (Nature Green, 2GB RAM, 32GB Storage) | 2GHz Octa-core Helio G25 Processor | 5000 mAh Battery
Redmi A1 (Black, 2GB RAM, 32GB Storage) | Segment Best AI Dual Cam | 5000mAh Battery | Leather Texture Design | Android 12
Redmi A1 (Light Green, 2GB RAM 32GB ROM) | Segment Best AI Dual Cam | 5000mAh Battery | Leather Texture Design | Android 12
realme narzo 50i (Mint Green, 4GB RAM+64GB Storage) Octa Core Processor | 6.5" inch Large Display
...

Answered By: AbiSaran

Why is the code not displaying total count of products?

Question:

Answers: