使用rvest设置cookies

Question

我想以编程方式导出this website上提供的记录。要手动执行此操作，我将导航到页面，单击导出，然后选择csv。

我试着从导出按钮复制链接，只要我有一个cookie (我相信)就可以工作。因此，wget或httr请求将导致html站点而不是文件。

我已经找到了some help from an issue on the rvest github repo，但最终我真的不能像问题制造者一样弄清楚如何使用对象来保存cookie并在请求中使用它。

这就是我的观点：

library(httr)
library(rvest)

apoc <- html_session("https://aws.state.ak.us/ApocReports/Registration/CandidateRegistration/CRForms.aspx")
headers <- headers(apoc)

GET(url = "https://aws.state.ak.us/ApocReports/Registration/CandidateRegistration/CRForms.aspx?exportAll=False&exportFormat=CSV&isExport=True", 
    add_headers(headers)) # how can I take the output from headers in httr and use it as an argument in GET from httr?

我已经检查了robots.txt，这是允许的。

Answer 1

您可以在获取https://aws.state.ak.us/ApocReports/Registration/CandidateRegistration/CRForms.aspx时从报头中获取__VIEWSTATE和__VIEWSTATEGENERATOR，然后在后续的POST查询和get csv中重用这些__VIEWSTATE和__VIEWSTATEGENERATOR。

options(stringsAsFactors=FALSE)
library(httr)
library(curl)
library(xml2)

url <- 'https://aws.state.ak.us/ApocReports/Registration/CandidateRegistration/CRForms.aspx'

#get session headers
req <- GET(url)
req_html <- read_html(rawToChar(req$content))
fields <- c("__VIEWSTATE","__VIEWSTATEGENERATOR")
viewheaders <- lapply(fields, function(x) {
    xml_attr(xml_find_first(req_html, paste0(".//input[@id='",x,"']")), "value")
})
names(viewheaders) <- fields

#post request. you can get the list of form fields using tools like Fiddler
params <- c(viewheaders,
    list(
        "M$ctl19"="M$UpdatePanel|M$C$csfFilter$btnExport",
        "M$C$csfFilter$ddlNameType"="Any",
        "M$C$csfFilter$ddlField"="Elections",
        "M$C$csfFilter$ddlReportYear"="2017",
        "M$C$csfFilter$ddlStatus"="Default",
        "M$C$csfFilter$ddlValue"=-1,
        "M$C$csfFilter$btnExport"="Export"))
resp <- POST(url, body=params, encode="form")
print(resp$status_code)
resptext <- rawToChar(resp$content)
#writeLines(resptext, "apoc.html")

#get response i.e. download csv
url <- "https://aws.state.ak.us//ApocReports/Registration/CandidateRegistration/CRForms.aspx?exportAll=True&exportFormat=CSV&isExport=True"
req <- GET(url, body=params)
read.csv(text=rawToChar(req$content))

您可能需要使用输入/代码来精确地获得您想要的东西。

下面是另一个使用RCurl的类似解决方案：how-to-login-and-then-download-a-file-from-aspx-web-pages-with-r