解析没有< table >/<td>/<tr>标记的表，数据嵌套在<div>标记中--漂亮汤、硒和webdriver_manager

Question

我正在尝试获取这个url = "https://www.topuniversities.com/university-rankings/university-subject-rankings/2021/psychology"“中的所有表。问题是没有table标记，也没有<tr>和<td>标记。行中的所有数据都在嵌套的"div”标记中。我使用的代码如下：

from bs4 import BeautifulSoup
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
import time

driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
driver.maximize_window()
driver.get(url)

time.sleep(5)
content = driver.page_source.encode('utf-8').strip()
soup = BeautifulSoup(content,"html.parser")

driver.quit()
print(soup)

另外，我只从嵌套<div>标记中的一个列(名为“总体得分”)获得数据。我还意识到，只有来自soup输出中的第10行的数据，但我正在尝试获取所有302行数据。

谢谢你给我的任何建议。

在的回答之后，我设法得到了预期的结果。这是我最后使用的代码：

res = requests.get('https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3519089_indicators.txt?1614801117').json()

df = pd.DataFrame(res["data"])
df = df[["uni", "region", "location", "city", "overall",
         "ind_69", "ind_70", "ind_76", "ind_77"]]
headers = {"uni":"University", "overall": "Overall Score", "ind_69": "H-index Citations",
           "ind_70": "Citations per Paper", "ind_76": "Academic Reputation", "ind_77": "Employer Reputation"}
df.rename(columns=headers, inplace=True)
for column in headers.values():
    df[column] = df[column].apply(lambda value: BeautifulSoup(value, 'html.parser').find('div').text)
df

DataFrame如下：

​

Answer 1

您不需要selenium，如果您转到network，您将得到以下链接，它以json的形式返回数据。您需要循环遍历它并获取值。

https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3519089.txt?1615516693?v=1616064930668

代码：

import requests
import json
res=requests.get("https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3519089.txt?1615516693?v=1616064930668").json()

print("Total records :{}".format(len(res['data'])))
for item in res['data']:
     print(item['country'])
     print(item['city'])
     print(item['score'])
     print("============")

输出：

Total records :302
United States
Cambridge
98.6
============
United States
Stanford
96.4
============
United Kingdom
Oxford
95.5
============
United Kingdom
Cambridge
94.8
============
United States
Berkeley
92.3
============
United States
Los Angeles
91.4
============
United States
New Haven
90.9
============
United States
Ann Arbor
89.5
============
United States
Cambridge
89.3
============
United Kingdom
London
89.2
============
United States
Philadelphia
89.2
============
United States
New York City
89.1
============
United States
New York City
88.4
============
United States
Chicago
88.2
============
Netherlands
Amsterdam
87.7
============
Singapore
Singapore
87.2
============
Canada
Vancouver
87.2
============
United States
Princeton
87
============
Canada
Toronto
86.1
============
United Kingdom
London
85.7
============
Australia
Parkville
85.7
============
United States
Evanston
85.5
============
Belgium
Leuven
85.2
============
United Kingdom
London
85.1
============
Australia
Sydney
85.1
============
Australia
Brisbane
84.4
============
Singapore
Singapore
84.3
============
United States
Durham
83.6
============
Canada
Montreal
83.5
============
Australia
Sydney
83.4
============
Netherlands
Utrecht
82.9
============
United States
Champaign
82.7
============
United Kingdom
Edinburgh
82.5
============
United Kingdom
Manchester
81.7
============
Hong Kong SAR
Hong Kong
81.7
============
United States
Austin
81.6
============
United States
Pittsburgh
81.5
============
Australia
Canberra
81.3
============
Netherlands
Rotterdam
81.2
============
United States
East Lansing
81.1
============
Germany
Berlin
81
============
Australia
Perth
81
============
Germany
Berlin
80.9
============
Netherlands
Groningen
80.9
============
United States
Ithaca
80.7
============
Hong Kong SAR
Hong Kong
80.4
============
United States
Madison
80.4
============
United States
Columbus
80.3
============
Switzerland
Zürich
80.3
============
United States
San Diego
80.2
============
Australia
Melbourne
80.1
============
Netherlands
Leiden
79.8
============
United States
Seattle
79.8
============
Netherlands
Tilburg
79.6
============
United States
Minneapolis
79.5
============
China (Mainland)
Beijing
79.4
============
New Zealand
Auckland
79.3
============
Netherlands
Maastricht
79.1
============
United States
University Park
79.1
============
United States
Chapel Hill
79.1
============
Belgium
Louvain-la-Neuve
78.9
============
Netherlands
Nijmegen
78.5
============
United Kingdom
Coventry
78.5
============
United States
Nashville
78.5
============
Netherlands
Amsterdam
78.5
============
United States
Baltimore
78.4
============
United Kingdom
Exeter
78.3
============
United States
College Park
78.3
============
United Kingdom
Cardiff
78.2
============
Germany
Munich
78.2
============
Chile
Santiago
78.1
============
New Zealand
Kelburn, Wellington
78.1
============
United States
Providence
78
============
Australia
Sydney
77.8
============
Belgium
Ghent
77.8
============
United States
Boston
77.3
============
United States
Los Angeles
77.3
============
Japan
Tokyo
77.1
============
United Kingdom
Birmingham
77.1
============
United Kingdom
Bristol
77
============
New Zealand
Dunedin
77
============
China (Mainland)
Beijing
76.9
============
Italy
Rome
76.9
============
Italy
Padua
76.9
============
United States
Charlottesville
76.9
============
Sweden
Stockholm
76.8
============
Spain
Madrid
76.8
============
United Kingdom
York
76.8
============
United States
Phoenix
76.6
============
Denmark
Aarhus
76.5
============ so on..

网络标签

​

​

​

​

Answer 2

我已经检查了您提供的URL。数据(从XHR请求@https://www.topuniversities.com/sites/default/files/qs-rankings-data/en/3519089.txt?1616049862?v=1616050007711接收)似乎是分页的，这就是为什么您只看到其中的10个条目。

您有两个选择来处理这个问题：

1. 模拟单击next页面按钮
2. 以JSON格式从XHR读取完整数据