从网上抓取数据

Question

我正试图从truecar.com中提取不同车型的价格、里程和型号。我想，我从网站上找到了合适的标签，但是我的代码没有给出合适的数据。我的代码给出了空数据，我没有发现问题。我怎么才能解决这个问题？

在定义价格、里程和型号时，我会收到“意想不到的缩进”。

import requests
from bs4 import BeautifulSoup
import mysql.connector
url='https://www.truecar.com/used-cars-for-sale/listings/'
r=requests.get(url)
soup=BeautifulSoup(r.text,'html.parser')

data = []
for card in soup.select('[class="card-content vehicle-card-body order-3 vehicle-card-carousel-
body"]'):
    price = card.select_one('[class="heading-3 margin-y-1 font-weight-bold"]').text

    miles = card.select_one('div[class="d-flex w-100 justify-content-between"]').text

    models=card.select_one('div[class="vehicle-card-top"]').text

    data.append({'price':price,'miles':miles,'models':models})

print(data)

Answer 1

您可以使用以下选择器来获取数据。

import requests
from bs4 import BeautifulSoup

url='https://www.truecar.com/used-cars-for-sale/listings/'
r=requests.get(url)

soup=BeautifulSoup(r.text,'html.parser')

cards = soup.select('div.linkable.card.card-shadow.vehicle-card._1qd1muk')

data = []
for card in cards:
    vehicleCardYearMakeModel = card.find("div", {"data-test" : "vehicleCardYearMakeModel"}).text.replace('Sponsored', '')
    vehicleMileage = card.find("div", {"data-test" : "vehicleMileage"}).text
    vehiclePrice = card.find("div", {"data-test" : "vehicleCardPricingBlockPrice"}).text
    data.append({'price':vehiclePrice,'miles':vehicleMileage,'models':vehicleCardYearMakeModel})

它给出了预期的输出。

[{'price': '$48,499', 'miles': '33,544 miles', 'models': '2018 BMW X5'}, {'price': '$19,900', 'miles': '76,418 miles', 'models': '2018 Chevrolet Equinox'}, {'price': '$48,795', 'miles': '40,551 miles', 'models': '2018 BMW 7 Series'}, {'price': '$14,445', 'miles': '169,206 miles', 'models': '2019 Nissan Rogue'}, {'price': '$21,174', 'miles': '22 miles', 'models': '2020 Kia Optima'}, {'price': '$9,680', 'miles': '126,555 miles', 'models': '2013 Honda Civic'}, {'price': '$19,300', 'miles': '21,467 miles', 'models': '2020 Toyota Corolla'}, {'price': '$22,000', 'miles': '35,453 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$10,562', 'miles': '214,543 miles', 'models': '2018 Chevrolet Equinox'}, {'price': '$17,997', 'miles': '51,278 miles', 'models': '2020 Nissan Altima'}, {'price': '$70,788', 'miles': '5,854 miles', 'models': '2022 Ford Super Duty F-250'}, {'price': '$21,500', 'miles': '15,230 miles', 'models': '2020 Chevrolet Equinox'}, {'price': '$12,999', 'miles': '148,605 miles', 'models': '2018 Ford Escape'}, {'price': '$30,683', 'miles': '51,273 miles', 'models': '2018 Ford F-150'}, {'price': '$10,849', 'miles': '85,539 miles', 'models': '2015 Nissan Altima'}, {'price': '$12,522', 'miles': '118,858 miles', 'models': '2014 Ford Fusion'}, {'price': '$18,995', 'miles': '65,156 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$19,497', 'miles': '61,376 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$17,700', 'miles': '53,012 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$51,775', 'miles': '15,204 miles', 'models': '2021 Toyota Tundra'}, {'price': '$16,977', 'miles': '67,389 miles', 'models': '2019 Dodge Grand Caravan'}, {'price': '$18,900', 'miles': '75,023 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$18,000', 'miles': '53,271 miles', 'models': '2020 Chevrolet Malibu'}, {'price': '$20,900', 'miles': '182,907 miles', 'models': '2018 Ford F-150'}, {'price': '$15,909', 'miles': '84,249 miles', 'models': '2020 Toyota Corolla'}, {'price': '$18,836', 'miles': '69,731 miles', 'models': '2019 Dodge Grand Caravan'}, {'price': '$9,995', 'miles': '119,941 miles', 'models': '2015 Chrysler 200'}, {'price': '$20,998', 'miles': '36,529 miles', 'models': '2017 Chevrolet Silverado 1500'}, {'price': '$6,800', 'miles': '158,894 miles', 'models': '2015 Nissan Altima'}, {'price': '$11,666', 'miles': '83,328 miles', 'models': '2014 Ford Fusion'}, {'price': '$7,670', 'miles': '64,000 miles', 'models': '2015 Ford Focus'}, {'price': '$16,995', 'miles': '180,663 miles', 'models': '2018 Ford F-150'}, {'price': '$20,990', 'miles': '138,684 miles', 'models': '2016 Ford F-150'}]

Answer 2

您无法得到任何结果的原因是，在第一个选择中的类名中有一个空格。

所以没有什么可迭代的，您的for循环也不会启动。修复这个问题，您将从其余代码中获得结果：

soup.select('[class="card-content vehicle-card-body order-3 vehicle-card-carousel-body"]

而不是：

soup.select('[class="card-content vehicle-card-body order-3 vehicle-card-carousel- body"]