scraping data from web

Question

I am trying to scrap prices, miles and models of different cars from truecar.com. I think, I find appropriate tags from the website but my code doesn’t give me the suitable data. my code give me empty data and I do not find the problem. May you please help me? I receive unexpected indent when defining price, miles and model.

import requests
from bs4 import BeautifulSoup
import mysql.connector
url='https://www.truecar.com/used-cars-for-sale/listings/'
r=requests.get(url)
soup=BeautifulSoup(r.text,'html.parser')

data = []
for card in soup.select('[class="card-content vehicle-card-body order-3 vehicle-card-carousel- 
body"]'):
    price = card.select_one('[class="heading-3 margin-y-1 font-weight-bold"]').text

    miles = card.select_one('div[class="d-flex w-100 justify-content-between"]').text

    models=card.select_one('div[class="vehicle-card-top"]').text

    data.append({'price':price,'miles':miles,'models':models})

print(data)

Asked By: Ali

||

Source

Answer 1

You can use the following selectors to fetch the data.

import requests
from bs4 import BeautifulSoup

url='https://www.truecar.com/used-cars-for-sale/listings/'
r=requests.get(url)

soup=BeautifulSoup(r.text,'html.parser')

cards = soup.select('div.linkable.card.card-shadow.vehicle-card._1qd1muk')

data = []
for card in cards:
    vehicleCardYearMakeModel = card.find("div", {"data-test" : "vehicleCardYearMakeModel"}).text.replace('Sponsored', '')
    vehicleMileage = card.find("div", {"data-test" : "vehicleMileage"}).text
    vehiclePrice = card.find("div", {"data-test" : "vehicleCardPricingBlockPrice"}).text
    data.append({'price':vehiclePrice,'miles':vehicleMileage,'models':vehicleCardYearMakeModel})

which gives you the expected output

[{'price': '$48,499', 'miles': '33,544 miles', 'models': '2018 BMW X5'}, {'price': '$19,900', 'miles': '76,418 miles', 'models': '2018 Chevrolet Equinox'}, {'price': '$48,795', 'miles': '40,551 miles', 'models': '2018 BMW 7 Series'}, {'price': '$14,445', 'miles': '169,206 miles', 'models': '2019 Nissan Rogue'}, {'price': '$21,174', 'miles': '22 miles', 'models': '2020 Kia Optima'}, {'price': '$9,680', 'miles': '126,555 miles', 'models': '2013 Honda Civic'}, {'price': '$19,300', 'miles': '21,467 miles', 'models': '2020 Toyota Corolla'}, {'price': '$22,000', 'miles': '35,453 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$10,562', 'miles': '214,543 miles', 'models': '2018 Chevrolet Equinox'}, {'price': '$17,997', 'miles': '51,278 miles', 'models': '2020 Nissan Altima'}, {'price': '$70,788', 'miles': '5,854 miles', 'models': '2022 Ford Super Duty F-250'}, {'price': '$21,500', 'miles': '15,230 miles', 'models': '2020 Chevrolet Equinox'}, {'price': '$12,999', 'miles': '148,605 miles', 'models': '2018 Ford Escape'}, {'price': '$30,683', 'miles': '51,273 miles', 'models': '2018 Ford F-150'}, {'price': '$10,849', 'miles': '85,539 miles', 'models': '2015 Nissan Altima'}, {'price': '$12,522', 'miles': '118,858 miles', 'models': '2014 Ford Fusion'}, {'price': '$18,995', 'miles': '65,156 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$19,497', 'miles': '61,376 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$17,700', 'miles': '53,012 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$51,775', 'miles': '15,204 miles', 'models': '2021 Toyota Tundra'}, {'price': '$16,977', 'miles': '67,389 miles', 'models': '2019 Dodge Grand Caravan'}, {'price': '$18,900', 'miles': '75,023 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$18,000', 'miles': '53,271 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$20,900', 'miles': '182,907 miles', 'models': '2018 Ford F-150'}, {'price': '$15,909', 'miles': '84,249 miles', 'models': '2020 Toyota Corolla'}, {'price': '$18,836', 'miles': '69,731 miles', 'models': '2019 Dodge Grand Caravan'}, {'price': '$9,995', 'miles': '119,941 miles', 'models': '2015 Chrysler 200'}, {'price': '$20,998', 'miles': '36,529 miles', 'models': '2017 Chevrolet Silverado 1500'}, {'price': '$6,800', 'miles': '158,894 miles', 'models': '2015 Nissan Altima'}, {'price': '$11,666', 'miles': '83,328 miles', 'models': '2014 Ford Fusion'}, {'price': '$7,670', 'miles': '64,000 miles', 'models': '2015 Ford Focus'}, {'price': '$16,995', 'miles': '180,663 miles', 'models': '2018 Ford F-150'}, {'price': '$20,990', 'miles': '138,684 miles', 'models': '2016 Ford F-150'}]

Answered By: Himanshuman

Answer 2

Issue cause you wont get any result is, that there is a whitespace in one of you class names in the first selection. So there is nothing to iterate and your for-loop wont start. Fix this and you will get your results from the rest of your code:

soup.select('[class="card-content vehicle-card-body order-3 vehicle-card-carousel-body"]

instead of:

soup.select('[class="card-content vehicle-card-body order-3 vehicle-card-carousel- body"]

Answered By: HedgeHog

scraping data from web

Question:

Answers: