BeautifulSoup scrape – Fail to retrieve product list

I am reaching out because I am having some troubles to adjust a piece of code that is supposed to scrape some information from Amazon product pages (title, url, product name, etc.). Classic stuff for scraping training 🙂

So I basically wrote through different function:

1. One function to generate the URL to scrape
2. One function to navigate across the different element & made the value extracts

At the end I just run my driver & beautifulsoup & launch the two functions.

However, the outcome is not what I expect. I’d like to retrieve an organized csv file with 1 row per product retrieved & each associated information into columns. Nevertheless, I always end up with 1 or 2 rows, but not all products from all pages.

I assume this is something coming from my soup along with the "for loop" that is not properly going through all items (though I can’t figure out what exactly).

I’d like to get your opinion about this, do you have any clues?

Thank you very much for the help

from bs4 import BeautifulSoup
from selenium import webdriver
import csv

#Function to generate URL with search KW & page nb
def get_url(search_term,page):
    template = 'https://www.amazon.co.uk/s?k={}&page='+str(page)
    search_term = search_term.replace(' ','+')
    url = template.format(search_term)

    return url

#Function to retrieve all data from the page
def extract_record(item):
    atag = item.h2.a
    
    #Retrieve product name
    description = atag.text.strip()
    
    #Retrieve product URL
    url = 'https://www.amazon.co.uk' + atag.get('href')
    
    #Retrieve sponsored status
    try:
        sponso_parent = item.find('span','s-label-popover-default')
        sponso = sponso_parent.find('span', {'class': 'a-size-mini a-color-secondary', 'dir': 'auto'}).text
    except AttributeError:
        sponso = 'No' 

    #Retrieve price info
    try:
        price_parent = item.find('span','a-price')
        price = price_parent.find('span','a-offscreen').text
    except AttributeError:
        return
    
    #Retrieve avg product rating
    try:
        rating = item.i.text
    except AttributeError:
        rating = ''
    
    #Retrieve review count (if monetary value, nill it due to missing value)
    try:
        review_count = item.find('span', {'class': 'a-size-base', 'dir': 'auto'}).text
    except AttributeError:
        review_count = ''
    
    if "£" in review_count or "€" in review_count or "$" in review_count:
        review_count = 0
    
    result = (url, description, sponso, price, rating,  review_count)
    
    return result
        
record_final = []

#Loop through page nb
for page in range(1,3):
    url = get_url('laptop',page)
    print(url)
    
    #Instantiate web driver & retrieve page content with BS (then loop through every product)
    driver = webdriver.Chrome('\Users\rapha\Desktop\chromedriver.exe')
    driver.get(url)
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    final_soup = soup.find_all('div',{'data-component-type': 's-search-result'})
    
    try:
        for item in final_soup:
            record = extract_record(item)
            if record:
                record_final.append(record)
    except AttributeError:
        print('error_record')
    
    driver.close()

with open('resultsamz.csv','w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(['url', 'description', 'sponso', 'price', 'rating','review_count'])
    writer.writerow(record_final)