让H2O使用Sparklyr时遇到麻烦

Question

我正试图让H2O和Sparklyr一起在我的星星团(纱线)上工作。

spark_version(sc) = 2.4.4我的spark集群正在运行V2.4.4

根据这页面，与我的spark兼容的版本为2.4.5，H2O版本为REL补丁版本3。然而，当我安装这个版本时，我会被提示将我的H2O安装更新到下一个版本( release )。在H2O指南和sparklyr指南之间，有时非常令人困惑和矛盾。

​

​

由于这是纱线部署，而不是本地的，不幸的是，我不能提供一个重复来帮助解决疑难问题。

url <- "http://h2o-release.s3.amazonaws.com/sparkling-water/rel-2.4/5/sparkling-water-2.4.5.zip"

download.file(url = url,"sparkling-water-2.4.5.zip")

unzip("sparkling-water-2.4.5.zip")

# RUN THESE CMDs FROM THE TERMINAL
cd sparkling-water-2.4.5
bin/sparkling-shell --conf "spark.executor.memory=1g"

# RUN THESE FROM WITHIN RSTUDIO
install.packages("sparklyr")
library(sparklyr)

# REMOVE PRIOR INSTALLS OF H2O
detach("package:rsparkling", unload = TRUE)
if ("package:h2o" %in% search()) { detach("package:h2o", unload = TRUE) }
if (isNamespaceLoaded("h2o")){ unloadNamespace("h2o") }
remove.packages("h2o")

# INSTALLING REL-ZORN (3.36.0.3) WHICH IS REQUIRED FOR SPARKLING WATER 3.36.0.3
install.packages("h2o", type = "source", repos = "https://h2o-release.s3.amazonaws.com/h2o/rel-zorn/3/R")

# INSTALLING FROM S3 SINCE CRAN NO LONGER SUPPORTED
install.packages("rsparkling", type = "source", repos = "http://h2o-release.s3.amazonaws.com/sparkling-water/spark-2.4/3.36.0.3-1-2.4/R")

# AS PER THE GUIDE
options(rsparkling.sparklingwater.version = "2.4.5")
library(rsparkling)

# SPECIFY THE CONFIGURATION
config <- sparklyr::spark_config()
config[["spark.yarn.queue"]] <- "my_data_science_queue"
config[["sparklyr.backend.timeout"]] <- 36000
config[["spark.executor.cores"]] <- 32
config[["spark.driver.cores"]] <- 32
config[["spark.executor.memory"]] <- "40g"
config[["spark.executor.instances"]] <- 8
config[["sparklyr.shell.driver-memory"]] <- "16g"
config[["spark.default.parallelism"]] <- "8"
config[["spark.rpc.message.maxSize"]] <- "256"

# MAKE A SPARK CONNECTION
sc <- sparklyr::spark_connect(
  master = "yarn",
  spark_home = "/opt/mapr/spark/spark",
  config = config,
  log = "console",
  version = "2.4.4"
)

当我尝试使用下一个块建立一个H2O上下文时，我会得到以下错误

h2o_context(sc)

Error in h2o_context(sc) : could not find function "h2o_context"

任何关于我哪里出错的建议都将不胜感激。

Answer 1

请看本教程。较新的版本使用{H2OContext.getOrCreate(h2oConf)}而不是{h2o_context(sc)}。