尝试获取Selenium以抓取表并单击按钮

Question

我将下面的代码合并在一起，试图将数据从HTML表中刮到数据帧，然后单击一个按钮移到下一页，但是它给了我一个错误，tat说“无效选择器”。

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from bs4 import BeautifulSoup
import time
from time import sleep
import pandas as pd


browser = webdriver.Chrome("C:/Utility/chromedriver.exe")
wait = WebDriverWait(browser, 10)

url = 'https://healthdata.gov/dataset/Hospital-Detail-Map/tagw-nk32'
browser.get(url)

for x in range(1, 5950, 13):
    time.sleep(3) # wait page open complete
    
    df = pd.read_html(browser.find_element_by_xpath("socrata-table frozen-columns").get_attribute('outerHTML'))[0]
    
    submit_button = browser.find_elements_by_xpath('pager-button-next')[0]
    submit_button.click()

我看到桌子了，但我不能参考。

​

​

知道这是怎么回事吗？

Answer 1

我已经用find_elements_by_css_selector找到了按钮

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from bs4 import BeautifulSoup
import time
from time import sleep
import pandas as pd

browser = webdriver.Chrome("C:/Utility/chromedriver.exe")
wait = WebDriverWait(browser, 10)

url = 'https://healthdata.gov/dataset/Hospital-Detail-Map/tagw-nk32'
browser.get(url)

for x in range(1, 5950, 13):
    time.sleep(3)  # wait page open complete

    df = pd.read_html(
        browser.find_element_by_xpath("socrata-table frozen-columns").get_attribute(
            'outerHTML'))[0]

    submit_button = browser.find_elements_by_css_selector('button.pager-button-next')[1]
    submit_button.click()

有时分页挂起，submit_button.click()以错误结束

selenium.common.exceptions.ElementClickInterceptedException: 
Message: element click intercepted: 
Element <button class="pager-button-next">...</button> 
is not clickable at point (182, 637). 
Other element would receive the click: <span class="site-name">...</span>

因此，考虑增加超时。例如，您可以使用以下方法

def click_timeout(element, timeout: int = 60):
    for i in range(timeout):
        time.sleep(1)
        try:
            element.click()
        except WebDriverException:
            pass
    element.click()

因此，单击元素的速度与其准备好的速度相同。