使用Selenium使用load more按钮抓取无限滚动的页面

Question

我试着通过网络抓取来获取所有项目的标题和创建者的名字，但是当我试图用“加载更多”按钮来抓取无限滚动的页面时，我得到了一个"TimeoutException: Message:“。请让我知道哪里错了，我需要改正什么。谢谢

下面是当前正在使用的代码：

from bs4 import BeautifulSoup
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome('/usr/local/bin/chromedriver')
driver.get("https://www.kickstarter.com/discover/advanced?sort=newest&seed=2695789&page=1/")

button = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,'bttn keyboard-focusable bttn-medium bttn-primary theme--create fill-bttn-icon hover-fill-bttn-icon')))
button.click()

names=[]
creators=[] 
soup = BeautifulSoup(driver.page_source)
for a in soup.findAll('div',{'class':'js-react-proj-card grid-col-12 grid-col-6-sm grid-col-4-lg'}):
    name=a.find('div', attrs={'class':'clamp-5 navy-500 mb3 hover-target'})
    creator=a.find('div', attrs={'class':'type-13 flex'})
    names.append(name.h3.text) 
    creators.append(creator.text)

df = pd.DataFrame({'Name':names,'Creator':creators}) 

Answer 1

你真的不需要使用Beautiful Soup和selenium。去找requests库吧，很容易就能免费获取所有的库。

import requests
import json
records = []
for i in range(5):
    req = requests.get('https://www.kickstarter.com/discover/advanced?google_chrome_workaround&woe_id=0&sort=newest&seed=2695910&page='+str(i),
                       headers={'Accept': 'application/json',
    'Content-Type': 'application/json'})
    if(req.status_code == 200):
        josn2 = req.json()
        projects = josn2.get("projects") 
        for i in range(len(projects)):
            print("Project Name - " + projects[i]['name'],end='          Created By - ')
            print(projects[i]['creator'].get('name'))

        print("----------------")

输出：

​

​

你可以向下滚动到页面，就像加载更多按钮加载内容一样，在for循环中放入足够的数值，你就会得到所有的内容。