MechanicalSoup不捕获所选数据

Question

我正在尝试使用以下代码。但我得到的数据是第一次选择日期的5倍。

import re
import datetime
import mechanicalsoup

def get_EminiTable(soup):
    allValues = []
    colnames = soup.find('thead').get_text()
    allValues.append([i for i in colnames.split('\n') if len(i) > 1])
    lnr = 0
    for line in soup.tbody.find_all('tr'):
        allValues.append([i for i in line.get_text().split('\n') if len(i) > 0])
        if 'UNCH' in allValues:
            allValues.pop()
        if lnr > 1: break
        lnr += 1
    return allValues

def get_settldays(soup):
    settlDays = soup.find('select', id="cmeTradeDate").get
    settlDays = re.findall('\d\d/\d\d/\d\d\d\d',str(settlDays))
    return [datetime.datetime.strptime(adat, '%m/%d/%Y') for adat in settlDays]


url =  "http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html"
browser = mechanicalsoup.StatefulBrowser()
if str(browser.open(url)) != '<Response [200]>':
    print('Error')
    quit()
soup = browser.get_current_page()
settlDays = get_settldays(soup)
for adate in settlDays:
    form = browser.select_form('form[id="quotesoptionsform1"]')
    form.set("tradeDate",adate.__format__('%m/%d/%Y'))
    browser.submit_selected()
    soup = browser.get_current_page()
    tabvals = get_EminiTable(soup)
    print(adate)
    for each in tabvals:
        print(each)
browser.session.close()
browser.close()

有没有办法获得不同表的正确日期，或者是机械汤中的bug。

Answer 1

我怀疑原因是因为这个表单是由JavaScript处理的，而不是HTTP，所以使用HTTP请求(这就是MechanicalSoup的工作方式)提交表单实际上不会做任何事情。

MechanicalSoup使用以下语法创建请求URL：

http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html?action=Submit&tradeDate=12%2F15%2F2017

如果单击此链接，您可以看到查询字符串&action=Submit&tradeDate=12%2F15%2F2017被忽略。然而，如果我们在真正的网络浏览器中手动设置交易日期，我们会看到它将#tradeDate=12/15/2017附加到URL。

即使我从web浏览器中获取该网址并使用MechanicalSoup打开它，它也不能正确显示，因为JavaScript未被处理。您可以通过以下方式修改最后一个for循环来查看：

for adate in settlDays:
    # Open the URL for each date directly
    date = adate.__format__('%m/%d/%Y')
    resp = browser.open(url + "#tradeDate={}".format(date))

    # Make sure we constructed the URL correctly
    print(resp.url)

    # Print the date that is being displayed
    soup = browser.get_current_page()
    print(soup.find('select', id='cmeTradeDate').find('option', attrs={'selected': 'selected'}).text)

输出如下所示(忽略语法突出显示...)：

http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/21/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/20/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/19/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/18/2017
Thursday, 21 Dec 2017 (Prelim)
http://www.cmegroup.com/trading/equity-index/us-index/e-mini-sandp500_quotes_settlements_futures.html#tradeDate=12/15/2017
Thursday, 21 Dec 2017 (Prelim)

除非有人知道使用HTTP处理此页面的替代方法，否则我认为您最好的选择是使用可以与JavaScript交互的工具，例如Selenium，它模拟真实的浏览器。