TimeoutException:消息: Stacktrace: Backtrace：

Question

我正在尝试从多个页面中进行web抓取，我的代码似乎对第一页非常有效，当我使用循环执行网页报废时--例如，前5页然后im获取以下错误:TimeoutException: Message: Stacktrace: Backtrace：

我的代码在下面

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys

import pandas as pd

from bs4 import BeautifulSoup

import requests as r


import time

from selenium.webdriver.support.ui import Select

PATH="chromedriver.exe"


driver=webdriver.Chrome(PATH)

_list=[]
for page_num in range(1,3):
    #print("----")
    url=f"https://valuebuds.com/pages/search-results-page?tab=products&page={page_num}"
    driver.get(url)
    
    Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#year_field")))).select_by_visible_text('1999')

    driver.find_element_by_class_name("agree").click()
    
    title=driver.find_elements_by_class_name("snize-overhidden")
    
    for j in title:
        Pro=j.find_element_by_class_name("snize-title").text
        Price=j.find_element_by_class_name("snize-price-list").text
        Desc=j.find_element_by_class_name("snize-description").text
    
        prec_item={
            "Product":Pro,
            "Price":Price,
            "Description":Desc
        }
        _list.append(prec_item)
        
df = pd.DataFrame(_list)

df.to_csv("Value Buds HTML Pricing.csv")

print("saved to file.")

请指点！提前感谢

Answer 1

代码块

Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#year_field")))).select_by_visible_text('1999')

driver.find_element_by_class_name("agree").click()

当你第一次登陆主页的时候。

一旦您选择了年份并单击了Agree按钮，您将能够看到显示结果的所有页面，而不需要再次选择该年。

所以，您的代码可能是这样的：

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys

import pandas as pd

from bs4 import BeautifulSoup

import requests as r


import time

from selenium.webdriver.support.ui import Select

PATH="chromedriver.exe"


driver=webdriver.Chrome(PATH)

_list=[]
for page_num in range(1,3):
    #print("----")
    url=f"https://valuebuds.com/pages/search-results-page?tab=products&page={page_num}"
    driver.get(url)

    if page_num == 1:
    
        Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "select#year_field")))).select_by_visible_text('1999')

       driver.find_element_by_class_name("agree").click()
    else:        
        time.sleep(2)
    
    title=driver.find_elements_by_class_name("snize-overhidden")
    
    for j in title:
        Pro=j.find_element_by_class_name("snize-title").text
        Price=j.find_element_by_class_name("snize-price-list").text
        Desc=j.find_element_by_class_name("snize-description").text
    
        prec_item={
            "Product":Pro,
            "Price":Price,
            "Description":Desc
        }
        _list.append(prec_item)
        
df = pd.DataFrame(_list)

df.to_csv("Value Buds HTML Pricing.csv")

print("saved to file.")

我已经为非第一次迭代添加了延迟，以便在您刮取它们的数据之前加载页面。

如果您在那里使用预期条件显式等待，我会更好。

我不知道在那里使用什么条件，留给你决定。