统计模型的形状未对齐错误GLM通过交叉验证预测

Question

我正在做一个问题，我在statsmodel中构建一个步长函数，同时首先使用交叉验证来确定理想的剪切量。然而，我遇到了一个问题，我就是不知道如何修复。

在我使用Sklearn的KFold函数添加了交叉验证循环后，我开始收到一个错误：

ValueError: shapes (480,2) and (1,) not aligned: 2 (dim 1) != 1 (dim 0)

我不太确定为什么现在会发生这种情况，因为我以前开始使用交叉验证循环，它工作得很好，没有任何问题。

如果有人能看一下我的代码块并指出这个问题的根源，我会非常感激。

在进入之前X_train和y_train的形状：

X_train: (2400,)  y_train: (2400,)

代码：

import statsmodels.api as sm
from sklearn.model_selection import KFold

kf = KFold(n_splits=5,shuffle=True, random_state=1)

cuts = []
RMSE = []

for i in range(1,11):
  cuts.append(i)
  cross_val_rms = []
  for train_index, test_index in kf.split(X_train):
    train_x,test_x= X_train.iloc[train_index], X_train.iloc[test_index]
    train_y,test_y= y_train.iloc[train_index], y_train.iloc[test_index]
    
    df_cut, bins = pd.cut(train_x, i, retbins=True, right=True)
    df_steps = pd.concat([train_x, df_cut, train_y],
                        keys=['age','age_cuts','wage'], axis = 1)
    df_steps_dummies = pd.get_dummies(df_cut)
    GLM_fitted = sm.GLM(df_steps.wage, df_steps_dummies).fit()
    bin_mapping = np.digitize(test_x, bins)
    X_valid = pd.get_dummies(bin_mapping)
    pred = GLM_fitted.predict(X_valid)
    rms = np.sqrt(mean_squared_error(test_y, pred))
    cross_val_rms.append(rms)
  mean_rms = sum(cross_vall_rms)/len(cross_vall_rms)
  RMSE.append(mean_rms)

cuts_df = pd.DataFrame()
cuts_df['Cuts'] = cuts
cuts_df['RMSE'] = RMSE

print('Cuts with lowest Root Mean Squared Error:',cuts_df.loc[cuts_df['RMSE'].idxmin], sep='\n')

错误：

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-166-a9794538c3e5> in <module>()
     21     bin_mapping = np.digitize(test_x, bins)
     22     X_valid = pd.get_dummies(bin_mapping)
---> 23     pred = GLM_fitted.predict(X_valid)
     24     rms = np.sqrt(mean_squared_error(test_y, pred))
     25     cross_val_rms.append(rms)

1 frames

/usr/local/lib/python3.7/dist-packages/statsmodels/genmod/generalized_linear_model.py in predict(self, params, exog, exposure, offset, linear)
    870             exog = self.exog
    871 
--> 872         linpred = np.dot(exog, params) + offset + exposure
    873         if linear:
    874             return linpred

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (480,2) and (1,) not aligned: 2 (dim 1) != 1 (dim 0)

Answer 1

我认为，如果你解释一下你在回归中试图做什么，这会有所帮助。你会得到错误，因为如果你从训练文件夹中得到3个箱子，这并不意味着你从测试文件夹中恰好得到了3个箱子，你可能会得到2个文件夹，因为在1个箱子中没有值。

在我看来，您可以简单地首先离散化这些值，然后使用示例数据进行训练：

import numpy as np
import pandas as pd
import statsmodels.api as sm
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import KFold


X_train = pd.Series(np.random.uniform(0,1,2400))
y_train = pd.Series(np.random.uniform(0,1,2400))

然后

for i in range(2,11):

  cross_val_rms = []
  df_steps_dummies = pd.get_dummies(pd.cut(X_train,i))
  
  for train_index, test_index in kf.split(X_train):
    train_x,test_x= df_steps_dummies.iloc[train_index,:], df_steps_dummies.iloc[test_index,:]
    train_y,test_y= y_train[train_index], y_train[test_index]
    
    GLM_fitted = sm.GLM(train_y, train_x).fit()
    pred = GLM_fitted.predict(test_x)
    rms = np.sqrt(mean_squared_error(test_y, pred))
    cross_val_rms.append(rms)

RMSE.append(np.array(cross_val_rms).mean())