不能从布局复杂的表格中抓取三个字段

Question

我在python和selenium中创建了一个脚本，可以从一个网站上的表中解析三个字段franking credit、gross divident和further information。最后两个字段只有在浏览器单击循环黄色按钮时才会显示，该按钮中有加号()。

但是，当单击按钮时，它们会变成红色，这表示信息已被显示。

我的脚本可以点击所有的按钮，但是它不能从那个表中抓取这三个字段。

我附上了一张图片，向你展示它的真实面貌。

我知道如果我向这个https://www.sharedividends.com.au/wp-content/custom/ajaxfile.php?code=MLT发送带有相关有效负载的post http请求，我可以得到json中的所有表格字段，但这不是我想要解决的问题。

网站链接

我试过：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.sharedividends.com.au/mlt-dividend-history/"

driver = webdriver.Chrome()

driver.get(url)

table = driver.find_element_by_css_selector("#divTable")
driver.execute_script("arguments[0].scrollIntoView();",table)

for items in driver.find_elements_by_css_selector("td.sorting_1"):
    driver.execute_script("arguments[0].scrollIntoView();",items)
    items.click()

for elems in driver.find_elements_by_css_selector("#divTable tbody tr"):
    franking_credit = elems.find_elements_by_css_selector("td")[5].text
    gross_divident = elems.find_elements_by_css_selector("td")[6].text
    further_info = elems.find_elements_by_css_selector("td")[7].text
    print(franking_credit,gross_divident,further_info)

driver.quit()

当我运行上面的脚本时，它会抛出这个错误，IndexError: list index out of range指向franking_credit =这一行。

这张桌子是这样的。我已经在我感兴趣的图片中标记了表中的三个字段。

https://filebin.net/x5adwz7y6hfir2sc

如何解析表中的三个字段？

Answer 1

您将得到以下错误，因为当运行自动化脚本时，它会显示带有其他属性的20行，而不是10 rows.Try，下面的代码如下所示。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.sharedividends.com.au/mlt-dividend-history/"

driver = webdriver.Chrome()

driver.get(url)

table = driver.find_element_by_css_selector("#divTable")
driver.execute_script("arguments[0].scrollIntoView();",table)

for items in driver.find_elements_by_css_selector("td.sorting_1"):
    driver.execute_script("arguments[0].scrollIntoView();",items)
    items.click()

for elems in driver.find_elements_by_css_selector("#divTable tbody tr[role='row']"):
    franking_credit = elems.find_elements_by_css_selector("td")[5].text
    gross_divident = elems.find_elements_by_css_selector("td")[6].get_attribute('textContent')
    further_info = elems.find_elements_by_css_selector("td")[7].get_attribute('textContent')
    print(franking_credit, gross_divident,further_info)

控制台上的输出：

$ 0.0446 $ 0.1486 10.4C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0107 $ 0.0357 2.5C FRANKED@30%; SP ECIAL; DRP SUSP

$ 0.0386 $ 0.1286 9C FRANKED @ 30%; DR P NIL DISCOUNT

$ 0.0437 $ 0.1457 10.2C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0377 $ 0.1257 8.8C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0429 $ 0.1429 10C FRANKED @ 30%; D RP NIL DISCOUNT

$ 0.0373 $ 0.1243 8.7C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0424 $ 0.1414 9.9C FRANKED @ 30%; DRP NIL DISCOUNT

$ 0.0373 $ 0.1243 8.7C FRANKED @ 30%; DRP

$ 0.0441 $ 0.1471 10.3C FR@30%;0.4C SP ECIAL;DRP;NIL DIS

Answer 2

这应该就行了！

from selenium import webdriver

driver = webdriver.Chrome('chromedriver/chromedriver.exe')

driver.get("https://www.sharedividends.com.au/mlt-dividend-history/")

for button in driver.find_elements_by_class_name("sorting_1"):
    button.click()

# Returns first part of the info
for item in driver.find_elements_by_xpath("//tr[@role='row']/td"):
    print(item.text)

# Returns second part of info
for a in driver.find_elements_by_xpath("//ul[@class='dtr-details']/li"):
        print(a.text)

输出；这

Answer 3

要从三个字段提取数据

• 代码块： 从selenium导入from驱动程序从executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')从selenium.webdriver.common.by导入WebDriverWait从selenium.webdriver.support导入WebDriverWait as EC chrome_options = webdriver.ChromeOptions() WebDriverWait驱动程序= webdriver.Chrome(options=chrome_options，executable_path=r‘C：\Utility\BrowserDrivers\chromedriver.exe’) driver.get("https://www.sharedividends.com.au/mlt-dividend-history/") driver.execute_script("arguments.scrollIntoView()；)((驱动程序，20).until(EC.visibility_of_element_located((By.CSS_SELECTOR，“table#divTable”))(在WebDriverWait(driver，table#divTable)"//table@aria-describedby='divTable_info'//tbody//tr/td@class='sorting_1'")))：elem.click() all_fc = [my_elem.get_attribute("innerHTML")表示my_elem in WebDriverWait(driver，5).until(EC.visibility_of_all_elements_located((By.XPATH，)"//table@aria-describedby='divTable_info'//tbody//tr//tdposition()=6")))] all_gd = [my_elem.get_attribute("innerHTML")表示my_elem in WebDriverWait(driver，5).until(EC.visibility_of_all_elements_located((By.XPATH，)"//table@aria-describedby='divTable_info'//tbody//tr//tdposition()=7")))] all_fi = [my_elem.get_attribute("innerHTML")表示my_elem in WebDriverWait(驱动程序，5).until(EC.visibility_of_all_elements_located((By.XPATH，"//table@aria-describedby='divTable_info'//tbody//tr@class='child'//li//span@class='dtr-data'")))]表示x，y，Z in zip(all_fc，all_gd，all_fi)：打印(x，y，z)
• 控制台输出： $ 0.0446 $ 0.1486 $ 10.4C盖印@ 30%；DRP零折扣$ 0.0107 $ 0.0357 $ 2.5C盖印@ 30%；特殊用途；DRP SUSP $ 0.0386 $ 0.1286 9C法郎@ 30%；P零折扣$ 0.0437 $ 0.1457 10.2C盖章@ 30%；DRP零折扣$ 0.0377 $ 0.1257 $0.1257 8.8C法郎@ 30%；DRP零折扣$ 0.0429 $ 0.1429 10C法郎@ 30%；D折扣$ 0.0373 $ 0.1243 8.7C法郎@30%；DRP零折扣$ 0.0424 $ 0.1414 9.9C盖印@ 30%；DRP零折扣$ 0.0373 $ 0.1243 8.7C盖印@ 30%；DRP $ 0.0441 $ 0.1471 $ 10.3C FR@30%;0.4C特殊用途；DRP；