Web scrapping gives different output every time

Question

from urllib import request
from bs4 import BeautifulSoup
page_url = "http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=-1&IsNodeId=1&Description=GTX&bop=And&Page=1&PageSize=36&order=BESTMATCH"
uclient = request.urlopen(page_url) #open a webclient
html_page = uclient.read()
page_soup = BeautifulSoup(html_page,"html.parser") 
uclient.close()
containers = page_soup.find_all("div",{"class" :"item-cell"}) 
title_list = []
for contain in containers:
    title = contain.select("img")[0]["title"]
    print(title)# for troubleshooting
    print(len(title_list)) #for troubleshooting
    title_list.append(title)
print(title_list)

Can someone pls help in troubleshooting? Everytime i run the code , once it returns 12 values sometimes 28 , sometimes 30 and then it gives an error .:

Input In [67], in <cell line: 16>()
     15 title_list = []
     16 for contain in containers:
---> 17     title = contain.select("img")[0]["title"]
     18     print(title)# for troubleshooting
     19     print(len(title_list)) #for troubleshooting

File C:ProgramDataAnaconda3libsite-packagesbs4element.py:1519, in Tag.__getitem__(self, key)
   1516 def __getitem__(self, key):
   1517     """tag[key] returns the value of the 'key' attribute for the Tag,
   1518     and throws an exception if it's not there."""
-> 1519     return self.attrs[key]

KeyError: 'title'

Asked By: aishwarya murali

||

Source

Answer 1

From your code, it looks like you’re trying to print out a list of products on the page. I find this code much better at returning those titles.

containers = page_soup.select(".item-title") 
titles = [c.text for c in containers if len(c.text) > 25]

I used the select method to find every class instance of ‘.item-title’ instead, and grabbed the text from that element for each line.

Sample output

MSI Ventus GeForce GTX 1660 SUPER 6GB GDDR6 PCI Express 3.0 x16 Video Card GTX 1660 SUPER VENTUS XS OC
EVGA GeForce GTX 1650 SC ULTRA GAMING, 04G-P4-1057-KR, 4GB GDDR5, Dual Fan, Metal Backplate
MSI Gaming GeForce GTX 1660 SUPER 6GB GDDR6 PCI Express 3.0 x16 Video Card GTX 1660 SUPER GAMING X
ASUS TUF Gaming GeForce GTX 1650 OC Edition 4GB GDDR6 PCI Express 3.0 Video Card TUF-GTX1650-O4GD6-P-GAMING
MSI Ventus GeForce GTX 1650 4GB GDDR6 PCI Express 3.0 x16 Video Card GTX 1650 D6 VENTUS XS
ASUS Dual GeForce RTX 3050 8GB GDDR6 PCI Express 4.0 Video Card DUAL-RTX3050-O8G

Answered By: Jackie

Web scrapping gives different output every time

Question:

Answers: