从多个页面中抓取表格

Question

我想从this网站上抓取数据。这个表扩展到多个页面上，但是url没有像page=3这样的东西，所以我不能遍历各个页面来获取数据。

有没有其他方法可以像这样抓取数据，而不会在不同的页面上循环？如何从python访问不同的页面来抓取数据？

Answer 1

使用@SeanBreckenridge提供的直接JSON链接，您可以创建一个包含所有数据的CSV文件，如下所示：

import requests
import json
import csv

header = ['University', 'Department', 'Overall Pay', 'LW Ratio', 'Academic Year', 'Program Year', 'Comment', 'V1', 'V2', 'V3', 'V4']

with open('output.csv', 'w', newline='') as f_output:
    csv_output = csv.writer(f_output)
    csv_output.writerow(header)

    for index in range(30):
        url = 'http://www.phdstipends.com/data/{}'.format(index)
        print(url)
        r = requests.get(url)

        if len(r.text) < 20:
            break

        csv_output.writerows(json.loads(r.text)['data'])

这将开始：

University,Department,Overall Pay,LW Ratio,Academic Year,Program Year,Comment,V1,V2,V3,V4
University of California - Santa Barbara (UCSB),Chemical Engineering,"$31,125",1.27,2014-2015,1st,,"$31,250",,,$125
Pennsylvania State University (Penn State),Chemistry,"$20,500",1.11,2014-2015,2nd,,"$20,500",,,
University of California - San Diego (UCSD),Mathematics,"$20,500",0.87,2013-2014,4th,,,"$18,000","$2,500",