Selenium只在少数几个页面上工作--元素在点上是不可点击的。

Question

我试图使用selenium自动生成Amazon/novels列表页面。有时起作用，有时不起作用。我无法理解代码中的错误。它工作了一段时间，滚动了20页中的13页，但从下一次开始，它就不能正常工作了。直到现在，它还没有滚动完成20页。

from selenium import webdriver
from time import sleep
from bs4 import BeautifulSoup


class App:

    def __init__(self,path='F:\Imaging'):

        self.path=path
        self.driver = webdriver.Chrome('F:\chromedriver')
        self.driver.get('https://www.amazon.in/s/ref=sr_pg_1?rh=i%3Aaps%2Ck%3Anovels&keywords=novels&ie=UTF8&qid=1510727563')
        sleep(1)
        self.scroll_down()
        self.driver.close()

    def scroll_down(self):

        self.driver.execute_script("window.scrollTo(0,5500);")
        sleep(1)
        load_more = self.driver.find_element_by_xpath('//span[@class="pagnRA"]/a[@title="Next Page"]')
        load_more.click()
        sleep(2)

        for value in range(2,19):

            print(self.driver.current_url)
            sleep(3)
            self.driver.execute_script("window.scrollTo(0,5500);")
            sleep(2)
            load_more = self.driver.find_element_by_xpath('//span[@class="pagnRA"]/a[@title="Next Page"]')
            load_more.click()

        sleep(3)


if __name__=='__main__':
    app=App()

我得到的代码的输出是：

    C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\python.exe C:/Users/Akhil/Scrape/amazon.py
https://www.amazon.in/s/ref=sr_pg_2/257-8503487-3570721?rh=i%3Aaps%2Ck%3Anovels&page=2&keywords=novels&ie=UTF8&qid=1510744188
https://www.amazon.in/s/ref=sr_pg_3?rh=i%3Aaps%2Ck%3Anovels&page=3&keywords=novels&ie=UTF8&qid=1510744197
https://www.amazon.in/s/ref=sr_pg_4?rh=i%3Aaps%2Ck%3Anovels&page=4&keywords=novels&ie=UTF8&qid=1510744204
Traceback (most recent call last):
  File "C:/Users/Akhil/Scrape/amazon.py", line 31, in <module>
    app=App()
  File "C:/Users/Akhil/Scrape/amazon.py", line 11, in __init__
    self.scroll_down()
  File "C:/Users/Akhil/Scrape/amazon.py", line 26, in scroll_down
    load_more.click()
  File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 501, in _execute
    return self._parent.execute(command, params)
  File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 308, in execute
    self.error_handler.check_response(response)
  File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element <a title="Next Page" id="pagnNextLink" class="pagnNext" href="/gp/search/ref=sr_pg_5?rh=i%3Aaps%2Ck%3Anovels&amp;page=5&amp;keywords=novels&amp;ie=UTF8&amp;qid=1510744210">...</a> is not clickable at point (809, 8). Other element would receive the click: <a href="/gp/prime/ref=nav_prime_try_btn/257-8503487-3570721" class="nav-a nav-a-2" data-ux-mouseover="true" id="nav-link-prime" tabindex="26">...</a>
  (Session info: chrome=62.0.3202.94)
  (Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 10.0.15063 x86_64)


Process finished with exit code 1

如何解决这个问题？

Answer 1

尝试以下代码：

load_more = ui.WebDriverWait(driver, timeout).until(EC.element_to_be_clickable((By.XPATH, "//span[@class="pagnRA"]/a[@title="Next Page"]")))
driver.execute_script("arguments[0].scrollIntoView(true);", load_more)
load_more.click()

超时-等待元素可点击的时间(以秒为单位)。

另外，在脚本的开头导入以下内容：

from selenium.webdriver.support import ui
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Answer 2

错误是没有下一个Page元素可见或可单击。您可以等待这个元素像这样的出现，或者将.click()放在一个try / exception块中，以检测它何时失败。

这可能是因为您的目标已经合法地用完了下一个页面(您已经看到了它们的全部)，或者页面仍然在加载，或者下一个链接的格式已经改变。

Answer 3

通过对@RatmirAsanov给出的答案的小修改，我终于得到了正确的结果。

请看这段代码。这将滚动所有页面，不会失败。

from selenium import webdriver
from time import sleep
from bs4 import BeautifulSoup
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

class App:
    def __init__(self,path='F:\Imaging'):
        self.path=path
        self.driver = webdriver.Chrome('F:\chromedriver')
        self.driver.get('https://www.amazon.in/s/ref=sr_pg_1?rh=i%3Aaps%2Ck%3Anovels&keywords=novels&ie=UTF8&qid=1510727563')
        sleep(1)
        self.scroll_down()
        self.driver.close()

    def scroll_down(self):
        sleep(3)
        self.driver.execute_script("window.scrollTo(0,5450);")
        sleep(3)
        load_more = self.driver.find_element_by_xpath('//span[@class="pagnRA"]/a[@title="Next Page"]')
        load_more.click()
        sleep(3)
        for value in range(2,19):
            print(self.driver.current_url)
            sleep(5)
            self.driver.execute_script("window.scrollTo(0,5500);")
            sleep(3)
            load_more = WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//span[@class='pagnRA']/a[@title='Next Page']")))
            self.driver.execute_script("arguments[0].click();", load_more)
            #load_more.click()
            sleep(3)
        sleep(3)


if __name__=='__main__':
    APP=App()