找不到正确的网页来抓取数据-- web抓取。

Question

我正试图为这个网站上的课程列表节省费用。

然而，我很难找到一个页面，在那里我可以看到所有的课程列表和价格。

我想出了下面的代码，这就给出了一道菜的价格：

import pandas as pd
import requests
url = "https://www.learningconnection.philips.com/en/course/pinnacle%C2%B3-advanced-planning-education"
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
price = soup.select_one("[class*='field-price'] .even").text
print(price)

如有任何帮助/建议，请见谅！

Answer 1

这里有一种方法可以在这个感兴趣的领域进行循环。使用bs4 4.7.1 +是为了访问:contains

import requests
from bs4 import BeautifulSoup as bs

base = 'https://www.learningconnection.philips.com'
url = f'{base}/en/catalog/profession/biomedical-engineers'
courses = []
results = []

with requests.Session() as s:
    r = s.get(url)
    soup = bs(r.content, 'lxml')
    links = [base + i['href'] for i in soup.select('h3 a')]
    for link in links:
        r = s.get(link)
        soup = bs(r.content, 'lxml')
        courses+=[i['href'] for i in soup.select('.title a')]
    for course in courses:
        r = s.get(course)
        soup = bs(r.content, 'lxml')
        price = soup.select_one('em:contains("Tuition:")')
        if price is None:
            price = 'Not listed'
        else:
            price = price.text.replace('\xa0',' ')
        result = {'Title':soup.select_one('#page-title').text.replace('\xa0',' ')
                 ,'Description': soup.select_one('.field-item p').text.replace('\xa0',' ')
                 ,'Price': price
                 , 'Url':course}
        results.append(result)

print(results)

Answer 2

通过将搜索锚定在项目的父包装上，您可以找到价格：

import requests
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.learningconnection.philips.com/en/course/pinnacle%C2%B3-advanced-planning-education').text, 'html.parser')
prices = [i.find_all('div', {'class':'field-item even'})[2].text for i in d.find_all('fieldset', {'class':' group-overview field-group-fieldset panel panel-default form-wrapper'})]

输出：

['5141.00']

Answer 3

使用关键字搜索项目，然后根据搜索结果获取所有urls。一旦你得到urls循环它。

from bs4 import BeautifulSoup
import requests
Search_key='pinnacle'
url = "https://www.learningconnection.philips.com/en/search/site/{}".format(Search_key)
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
urls=[item['href'] for item in soup.select('h3.title > a')]
price=[]
for url in urls:
    soup = BeautifulSoup(requests.get(url).text, 'html.parser')
    if soup.select_one("[class*='field-price'] .even"):
        price.append(soup.select_one("[class*='field-price'] .even").text)
print(price)

输出：

['5171.00', '5171.00', '3292.00', '5141.00', '4309.00', '2130.00', '2130.00', '2130.00']

您还可以打印课程标题。

from bs4 import BeautifulSoup
import requests
Search_key='pinnacle'
url = "https://www.learningconnection.philips.com/en/search/site/{}".format(Search_key)
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
urls=[item['href'] for item in soup.select('h3.title > a')]
price=[]
title=[]
for url in urls:
    soup = BeautifulSoup(requests.get(url).text, 'html.parser')
    if soup.select_one("[class*='field-price'] .even"):
        title.append(soup.select_one("h1#page-title").text)
        price.append(soup.select_one("[class*='field-price'] .even").text)
print(title)
print(price)

输出：

['Pinnacle³ Auto Segmentation with SPICE', 'Pinnacle³ Dynamic Planning', 'Pinnacle³ Additional Education', 'Pinnacle³ Advanced Planning Education', 'Pinnacle³ Basic Planning Education', 'Pinnacle³ Physics Modeling', 'Pinnacle³ Level I Basic Planning Education', 'Pinnacle³ Level II Education']
['5171.00', '5171.00', '3292.00', '5141.00', '4309.00', '2130.00', '2130.00', '2130.00']

编辑的

from bs4 import BeautifulSoup
import requests
Search_key='biomed'
url = "https://www.learningconnection.philips.com/en/search/site/{}".format(Search_key)
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
urls=[item['href'] for item in soup.select('h3.title > a')]
print(len(urls))
price=[]
title=[]
for url in urls:
    soup = BeautifulSoup(requests.get(url).text, 'html.parser')
    if soup.select_one("[class*='field-price'] .even"):
        title.append(soup.select_one("h1#page-title").text)
        price.append(soup.select_one("[class*='field-price'] .even").text)
print(title)
print(price)

输出

28
['NETWORK CONCEPTS (BIOMED)']
['4875.00']