mlr3:在优化模型(即AutoTuner对象)中使用benchmark()

Question

我想比较几种机器学习算法的性能(例如，rpart、xgb、.的决策树)。包括他们使用mlr3进行的超参数调整。换句话说，我想比较已经调优的不同算法的实例，而不是算法的默认超参数值。

mlr3提供自动调谐器对象来执行嵌套重采样和超参数调优.还有一个基准()函数来对几个学习者进行比较。benchmark()函数依次使用benchmark_grid()来设置基准测试。根据本手册，可以将“AutoTuner对象传递给mlr3 3：：resample()或mlr3 3：：AutoTuner()”。我不明白如何将AutoTuner对象传递给benchmark_grid()。

以下代码(根据这本书中的代码对优化的决策树进行默认版本的基准测试)不起作用。它返回一个错误消息：" error :在DictionaryLearner中找不到键‘DictionaryLearner’的元素！“

library("mlr3verse")

### Benchmarking including hyperparameter tuning

# nested resampling:
# - inner sampling: 5-fold CV
# - outer sampling: manually defined hold-out sample

# defining AutoTuner for the inner resampling
learner = lrn("classif.rpart")
resampling = rsmp("cv", folds = 5)
# resampling = rsmp("holdout")
measure = msr("classif.acc")
search_space = ps(maxdepth = p_int(lower = 1, upper = 10))
terminator = trm("none")
tuner = tnr("grid_search", resolution = 5)

rpart_tuned = AutoTuner$new(learner, resampling, measure, terminator, tuner, search_space)

## Outer re-sampling
# hold-out sample with pre-defined partitioning into train and test set
outer_resampling = rsmp("custom")
train_sets = list(1:120)
test_sets = list(121:150)
outer_resampling$instantiate(task, train_sets, test_sets)

## Defining benchmark design

design = benchmark_grid(
  tasks = tsks(c("iris")),
  learners = lrns(c("rpart_tuned", "classif.rpart"),
                  predict_type = "prob", predict_sets = c("train", "test")),
  resamplings = outer_resampling
)

Answer 1

代码中的问题是，您正在尝试创建一个新的学习者，而不是在

lrns(c("rpart_tuned", "classif.rpart"),
                  predict_type = "prob", predict_sets = c("train", "test")),

lrns(c("rpart_tuned"))正试图从mlr3内置的学习者词典中检索一个rpart_tuned。

如果您想重用rpart_tuned，只需这样做：

design = benchmark_grid(
  tasks = tsks(c("iris")),
  learners = c(rpart_tuned, lrn("classif.rpart",
                  predict_type = "prob", predict_sets = c("train", "test"))),
  resamplings = outer_resampling
)

这反过来使用rpart_tuned自动调谐器并从字典中创建一个新的学习classif.rpart。