抓取堆栈溢出用户数据

Question

import requests
from bs4 import BeautifulSoup
import csv

response = requests.get('https://stackoverflow.com/users?page=3&tab=reputation&filter=week').text
soup = BeautifulSoup(response, 'lxml')
for items in soup.select('.user-details'):
    name = items.select("a")[0].text
    location = items.select(".user-location")[0].text
    reputation = items.select(".reputation-score")[0].text
    print(name,location,reputation)

    with open('stackdata.csv','a',newline='') as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow([name,location,reputation])

当我们改变这段代码的url时，输出保持不变。

Answer 1

我遇到了一个类似的问题。对我来说有效的解决方案是使用selenium。虽然我使用的是无头浏览器，即phantomjs，但我认为它也适用于其他浏览器。

driver = webdriver.PhantomJS('/home/practice/selenium/webdriver/phantomjs/bin/phantomjs')
users = []
page_num = 1
driver.get('https://stackoverflow.com/users?page={page_num}&tab=reputation&filter=week'.format(page_num=page_num))
content = driver.find_element_by_id('content')

for details in content.find_elements_by_class_name('user-details'):
    users.append(details.text)

print(users)

更改page_num以获得所需的结果。

希望这能有所帮助！