网络抓取？

Question

我正在做以下网站上的一项网络抓取作业：

http://gomason.com/schedule.aspx?path=mbball

我被要求计算男子队(梅森)在赢得一场比赛和输掉一场比赛时的平均得分。

每场比赛的得分以梅森的得分-对手得分的格式列在每行的右侧。例如，第一个上市游戏的得分为67-65分。通过使用SelectorGadget工具，我确定了刮取这些信息所需的CSS选择器，然后编写了代码来抓取这些信息。这个向量中有33条数据。我用分离()函数把梅森的分数和对手的分数分开。

我尝试过几种方法，但它给了我以下错误：

mens_bb <- read_html("http://gomason.com/schedule.aspx?path=mbball", na = c("NA")) 

mens_scores <- mens_bb %>%
html_nodes("div.sidearm-schedule-game-result span:nth-child(3)") %>%
html_text()
as.numeric()
as.double()

mens_df <- data_frame(
  date = mens_dates, time = mens_times, opponent = mens_opponents, location = mens_locations, score = mens_scores, win_loss = mens_win_loss)

mens_df$score <- as.numeric(mens_df$score)       
mens_df$score <- as.double(mens_df$score)

NAs introduced by coercion

mens_df %>% separate(score, c("Mason’s score", "Opponent’s score"), sep = "\\-") 

mens_average <- mean(score, na = c("NA")) 

argument is not numeric or logical: returning NA

它列出了现在所有的分数为NAs，变量分数保持为chr类型。我怎么能解决这个问题？

在分离分数值之前，mens_df是这样的：

Observations: 33
Variables: 6
$ date     <chr> "Nov 10 (Fri)", "Nov 12 (Sun)", "Nov 16 (Thu)", "No...
$ time     <chr> "7:00 p.m. ", "2:00 p.m. ", "7:00 p.m. ", "6:00 p.m...
$ opponent <chr> "Lafayette ", "Louisville", "Binghamton ", "CSUN", ...
$ location <chr> "Fairfax, Va.", "Louisville, Ky.", "Fairfax, Va. ",...
$ score    <chr> "67-65", "61-72", "69-57", "78-73", "64-77", "73-79...
$ win_loss <chr> "W", "L", "W", "W", "L", "L", "W", "L", "L", "W", "...

Answer 1

library(rvest)
library(tidyverse)

# read webpage/ extract only container with relevant info 
bb_mason <- read_html("http://gomason.com/schedule.aspx?path=mbball")  %>% 
            xml_find_all(".//ul[@class = 'sidearm-schedule-games-container']")

# vector with xpath to extract required info
bb_xpath <- c(opponent = ".//span[@class = 'sidearm-schedule-game-opponent-name']/a",
       location = ".//div[@class = 'sidearm-schedule-game-location']/span[1]",
       result = ".//div[@class = 'sidearm-schedule-game-result text-italic']/span[2]",
       score = ".//div[@class = 'sidearm-schedule-game-result text-italic']/span[3]")

# sapply all xpath values, fetch information, and store in data frame tibble
bb_df <- tbl_df(
             sapply(bb_xpath, function(x) { 
                                    bb_mason %>% 
                                      xml_find_all(x) %>%
                                      xml_text(trim = T) }))

# separate scores, replace values in result column with more appropriate values,
# convert to numeric, and calculate average
bb_df %>% 
  separate("score", c("mason", "opp"), sep = "-") %>% 
  mutate(mason = as.numeric(mason), 
         opp = as.numeric(opp),
         result = plyr::mapvalues(result, c("W,", "L,"), c("Won", "Lost"))) %>% 
  group_by(result) %>% 
  summarize(avg.score = mean(mason))

最后产出：

#  A tibble: 2 x 2
#   result avg.score
#   <chr>      <dbl>
# 1 Lost        65.7
# 2 Won         78.4

Answer 2

您调用as.numeric()太早了，因为分数仍然包含一个破折号('-')。这就是为什么你要得到NA的原因。

您可以这样做：(我已经创建了一个只包含分数的新的dataframe )。

library(rvest)
mens_bb <- read_html("http://gomason.com/schedule.aspx?path=mbball", na = c("NA")) 

mens_scores <- mens_bb %>%
  html_nodes("div.sidearm-schedule-game-result span:nth-child(3)") %>%
  html_text()

library(tidyr)
mens_df1 <- data.frame(mens_scores)
mens_df1 <- mens_df %>% separate(mens_scores, c("Mason_score", "Opponent_score"), sep = "\\-")
mens_df1$Mason_score <- as.numeric(mens_df1$Mason_score)
mens_average <- mean(mens_df1$Mason_score)
#71.84848

Answer 3

您应该在转换到numeric之前进行分离。例如，下面的内容将给出您想要的结果。

mens_df <- data.frame(mens_scores)
mens_df %>% separate(mens_scores, c("Mason’s score", "Opponent’s score"), sep = "-") %>%
  mutate_all(as.numeric) %>% summarise_all(mean)

#   Mason’s score Opponent’s score
# 1         71.85            75.67