尝试使用MLPClassifier hidden_layer_sizes优化BayesSearchCV时出错

Question

当尝试调优sklearn BayesSearchCV，MLPClassifier hidden_layer_sizes超级参数时，我会得到一个错误：ValueError: can only convert an array of size 1 to a Python scalar。

然而，当我使用GridSearchCV时，它工作得很好！，我遗漏了什么？

下面是一个可重复的例子：

from skopt import BayesSearchCV
from skopt.space import Real, Categorical, Integer
from sklearn.datasets import load_iris
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV

X, y = load_iris(True)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                 train_size=0.75,
                                                 random_state=0)
# this does not work!
opt_bs = BayesSearchCV(MLPClassifier(), 
                     {'learning_rate_init': Real(0.001, 0.05),
                        'solver': Categorical(["adam", 'sgd']), 
                        'hidden_layer_sizes': Categorical([(10,5), (15,10,5)])}, 
                     n_iter=32,
                     random_state=0)

# this one does :)
opt_gs = GridSearchCV(MLPClassifier(), 
                   {'learning_rate_init': [0.001, 0.05],
                        'solver': ["adam", 'sgd'], 
                        'hidden_layer_sizes': [(10,5), (15,10,5)]})
                   
# executes optimization using opt_gs or opt_bs
opt = opt_bs
res = opt.fit(X_train, y_train)
opt

生产：

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-64-78e6d29cae99> in <module>()
     27 # executes optimization using opt_gs or opt_bs
     28 opt = opt_bs
---> 29 res = opt.fit(X_train, y_train)
     30 opt

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in fit(self, X, y, groups, callback)
    678                 optim_result = self._step(
    679                     X, y, search_space, optimizer,
--> 680                     groups=groups, n_points=n_points_adjusted
    681                 )
    682                 n_iter -= n_points

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in _step(self, X, y, search_space, optimizer, groups, n_points)
    553 
    554         # convert parameters to python native types
--> 555         params = [[np.array(v).item() for v in p] for p in params]
    556 
    557         # make lists into dictionaries

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in <listcomp>(.0)
    553 
    554         # convert parameters to python native types
--> 555         params = [[np.array(v).item() for v in p] for p in params]
    556 
    557         # make lists into dictionaries

/usr/local/lib/python3.6/dist-packages/skopt/searchcv.py in <listcomp>(.0)
    553 
    554         # convert parameters to python native types
--> 555         params = [[np.array(v).item() for v in p] for p in params]
    556 
    557         # make lists into dictionaries

ValueError: can only convert an array of size 1 to a Python scalar

Answer 1

不幸的是，BayesSearchCV只接受分类、整数或实类型值中的参数。在您的示例中，不存在w.r.t learning_rate_init和solver参数的问题，因为它们分别明确定义为Real和Categorical，问题出现在hidden_layer_sizes中，您已经将神经元的数量声明为Categorical值，在本例中为元组，而BayesSearchCV尚未准备好处理元组中的搜索空间，有关此问题的更多细节，请参阅这里。但是，作为一种临时攻击，您可以在MLPClassifier周围创建自己的包装器，以便正确识别估计器的参数。有关示例，请参阅以下代码片段：

from skopt import BayesSearchCV
from skopt.space import Integer
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor, MLPClassifier
from sklearn.base import BaseEstimator, ClassifierMixin
import itertools

X, y = load_iris(True)
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                 train_size=0.75,
                                                 random_state=0)

class MLPWrapper(BaseEstimator, ClassifierMixin):
    def __init__(self, layer1=10, layer2=10, layer3=10):
        self.layer1 = layer1
        self.layer2 = layer2
        self.layer3 = layer3

    def fit(self, X, y):
        model = MLPClassifier(
            hidden_layer_sizes=[self.layer1, self.layer2, self.layer3]
        )
        model.fit(X, y)
        self.model = model
        return self

    def predict(self, X):
        return self.model.predict(X)

    def score(self, X, y):
        return self.model.score(X, y)


opt = BayesSearchCV(
    estimator=MLPWrapper(),
    search_spaces={
        'layer1': Integer(10, 100),
        'layer2': Integer(10, 100),
        'layer3': Integer(10, 100)
    },
    n_iter=11
)

opt.fit(X_train, y_train)
opt.score(X_test,y_test)
0.9736842105263158

注意:这假设您构建了一个包含三个层的MLP网络。你可以根据你的需要修改它。另外，创建一个具有任意层数的MLP的类会变得有点棘手。