RSelenium将刮起雅虎财经新闻的头条

Question

我想从雅虎得到一家公司的新闻标题。我使用RSelenium启动远程浏览器并接受cookies。我找到了替代的css类"StretchedBox“，并且我可以通过浏览器检查从字面上看到标题。如何存储这些标题？接下来，我想使用RSelenium向下滚动并保存更多这些元素(比如保存几天)。

library('RSelenium')

# Start Remote Browser
rD <- rsDriver(port = 4840L, browser = c("firefox")) 
remDr <- rD[["client"]]

# Navigate to Yahoo Finance News for Specific Company
# This takes unusual long time 
remDr$navigate("https://finance.yahoo.com/quote/AAPL/news?p=AAPL")

# Get "accept all cookies" botton
webElems <- remDr$findElements(using = "xpath", "//button[starts-with(@class, 'btn primary')]") 

# We can check if we did get the proper button by checking the text of the element:
unlist(lapply(webElems, function(x) {x$getElementText()}))

# We found the two button, and we want to click the first one:
webElems[[1]]$clickElement()

# wait for page loading
Sys.sleep(5) 

# I am looking for news headline in or after the StretchedBox
boxes <- remDr$findElements(using = "class", "StretchedBox")
boxes[1] # empty

boxes[[1]]$browserName  

​

​

Answer 1

最后，我找到了一个可以用来getElementText新闻文章标题的xpath。

library('RSelenium')

# Start Browser
rD <- rsDriver(port = 4835L, browser = c("firefox")) 
remDr <- rD[["client"]]

# Navigate to Yahoo Financial News
remDr$navigate("https://finance.yahoo.com/quote/AAPL/news?p=AAPL")

# Click Accept Cookies
webElems <- remDr$findElements(using = "xpath", "//button[starts-with(@class, 'btn primary')]")
unlist(lapply(webElems, function(x) {x$getElementText()}))
webElems[[1]]$clickElement()

# extract headlines from html/css by xpath 
headlines <- remDr$findElements(using = "xpath", "//h3[@class = 'Mb(5px)']//a")

# extract headline text
headlines <- sapply(headlines, function(x){x$getElementText()})
headlines[1]

[[1]]
[1] "What Kind Of Investors Own Most Of Apple Inc. (NASDAQ:AAPL)?"