When using BeautifulSoup to scrape Ebay, I get the wrong price for a listing

Question

I am using BeautifulSoup to scrape the eBay site, however I got a different price than from the actual eBay page. I want to search for multiple different products later on and get the price for each of them, that’s why I use search_term.

Here is my code:

from bs4 import BeautifulSoup
import requests

# Set the search term
search_term = 'Ryco Air Filter A1785'

# Make a GET request to the eBay search page with the search term as a parameter
response = requests.get('https://www.ebay.com.au/sch/i.html', params={'_nkw': search_term})

# Parse the response content with BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')

# Find the price of the first item
print(soup.find("span", attrs={"class": "s-item__price"}))

I get this output:

<span class="s-item__price">$20.00</span>

However on the eBay page it says the product is $22.84.

Asked By: Behnam Mozafari

||

Source

Answer 1

The $20 span is also on the page but inside a hidden element

If you want the price from the first search result, I suggest using .select_one^{(view example)} like

print(soup.select_one('ul.srp-results li span.s-item__price'))

which is effectively the same as

print(soup.find(lambda s: s.name=='span' and 's-item__price' in s.get('class',[]) and s.find_parent(lambda p: p.name=='li' and p.find_parent('ul', {'class':'srp-results'}))))

[but select is obviously much simpler].

Answered By: Driftr95

When using BeautifulSoup to scrape Ebay, I get the wrong price for a listing

Question:

Answers: