从下拉列表组合中提取所有链接

Question

我有一个样本网站，我想从网站上提取所有的"href链接“。它有两个下拉列表，一旦选择了下拉列表，它就会显示结果，并链接到要下载的手册。它不会导航到不同的页面，而是在同一页面上显示结果。我已经提取了下拉列表的组合，我正在尝试提取手册链接，但我无法找到链接。

代码如下

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
from bs4 import BeautifulSoup
import requests


url = "https://www.cars.com/"

driver = webdriver.Chrome('C:/Users/webdrivers/chromedriver.exe')
driver.get(url)
time.sleep(4)

selectYear = Select(driver.find_element_by_id("odl-selected-year"))

data = []
for yearOption in selectYear.options:
    yearText = yearOption.text
    selectYear.select_by_visible_text(yearText)
    time.sleep(1)

    selectModel = Select(driver.find_element_by_id("odl-selected-model"))
    for modelOption in selectModel.options:
        modelText = modelOption.text
        selectModel.select_by_visible_text(modelText)
        data.append([yearText,modelText])

page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.text, 'html.parser')

content = soup.findAll('div',attrs={"class":"odl-results-container"})

for i in content:
    x = i.findAll(['h3','span'])
    for y in x:
        print(y.get_text())

打印不显示任何数据。我怎样才能获得手册的链接？提前感谢

Answer 1

您需要为每个汽车型号和年份单击按钮，然后单击和，然后从Selenium webdriver检索呈现的HTML页面源代码，而不是通过请求。

在你的内部循环中添加以下代码：

button = driver.find_element_by_link_text("Select this vehicle")
button.click()

page = driver.page_source
soup = BeautifulSoup(page, 'html.parser')
content = soup.findAll('a',attrs={"class":"odl-download-link"})
for i in content:
    print(i["href"])

这将打印出来：

http://www.fordservicecontent.com/Ford_Content/vdirsnet/OwnerManual/Home/Index?Variantid=6875&languageCode=EN&countryCode=USA&marketCode=US&bookcode=O91668&VIN=&userMarket=GBR
http://www.fordservicecontent.com/Ford_Content/vdirsnet/OwnerManual/Home/Index?Variantid=7126&languageCode=EN&countryCode=USA&marketCode=US&bookcode=O134871&VIN=&userMarket=GBR
http://www.fordservicecontent.com/Ford_Content/vdirsnet/OwnerManual/Home/Index?Variantid=7708&languageCode=EN&countryCode=USA&marketCode=US&bookcode=O177941&VIN=&userMarket=GBR
...