网页打印的标题为None，BeautifulSoup

Question

我正试图从本网站中抓取数据，却无法获得网页的标题。

我的密码-

import requests
from bs4 import BeautifulSoup

base_url = "https://www.stfrancismedicalcenter.com/find-a-provider/"

content = requests.get(url = base_url).content
soup = BeautifulSoup(content, "html.parser")

profile_link = soup.find("a", {"class": "flex-top-between-block-500"}).get("href")
profile_url = base_url + profile_link[1:]

profile_content = requests.get(url = profile_url).content
profile_soup = BeautifulSoup(profile_content, "html.parser")
print(profile_soup.title.string)

这是输出的结果。

[Running] python -u "d:\Personal\CS\Web Scrapping\first.py"
None

[Done] exited with code=0 in 3.592 seconds

我想就此提出一些建议！

Answer 1

这里的问题是，连接到配置文件的路径是不正确的，部分find-a-provider是重复的，因此它变成：

https://www.stfrancismedicalcenter.com/find-a-provider//find-a-provider/adegbenga-a-adetola-md/

相反，使用url定义一个特定的""baseUrl：

profile_url = 'https://www.stfrancismedicalcenter.com' + profile_link 

或

baseUrl = 'https://www.stfrancismedicalcenter.com'
profile_url =  baseUrl + profile_link 

示例

import requests
from bs4 import BeautifulSoup

url = "https://www.stfrancismedicalcenter.com/find-a-provider"
baseUrl = 'https://www.stfrancismedicalcenter.com'

content = requests.get(url).content
soup = BeautifulSoup(content, "html.parser")

profile_link = soup.find("a", {"class": "flex-top-between-block-500"}).get("href")
profile_url = baseUrl + profile_link

profile_content = requests.get(url = profile_url).content
profile_soup = BeautifulSoup(profile_content, "html.parser")
profile_soup.title.text

输出

Adegbenga A. Adetola MD