BeautifulSoup findAll not returning results

Question:

I want to get the product name and prices of this page. I pretty much repeated the exact same thing, I did for the product name for the price, but I’m not getting anything.

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as bSoup

header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0'}
url = "https://www.walmart.ca/search?q=lettuce"
req = Request (url = url, headers = header)

client = urlopen (req)
pageHtml = client.read()
client.close()

pageSoup = bSoup(pageHtml, 'html.parser')

products = pageSoup.findAll ("div", {"class":"css-155zfob e175iya63"})
print (len(products)) #prints 15, like expected
for product  in products:
    pass

prices = pageSoup.findAll ("div", {"class":"css-8frhg8 e175iya65"})
print (len(prices)) #prints 0 and idk why :/
for price in prices:
    pass
Asked By: skushu

||

Answers:

The page https://www.walmart.ca/search?q=lettuce does not return the content you expect:

curl -s -H 'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0' 'https://www.walmart.ca/search?q=lettuce' | grep 'css-8frhg8'

You probably saw that class in a browser where the content was partially render at run-time via JavaScript. This means you need to use a library that can emulate a browser with JavaScript support.

Answered By: Allan Wind
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.