当我使用imblearn管道而不是sklearn管道时，所有的文本特征都消失了。有解决办法吗？

Question

这是我下面的代码，我需要使用SMOTENC来平衡数据集，这意味着我必须使用来自imblearn库的管道。但是，它不承认CountVectorizer特性。

from imblearn.pipeline import Pipeline 
# from sklearn.pipeline import Pipeline

 vectorizer_params = dict(ngram_range=(1, 2), min_df=200, max_df=0.8)

 categorical_features = ['F1','F2','F3','F4']
 categorical_transformer = OneHotEncoder(handle_unknown="ignore")

 textual_feature = ['F5']
 text_transformer = Pipeline(
    steps=[
        ("squeez", FunctionTransformer(lambda x: x.squeeze())),
        ("vect", CountVectorizer(**vectorizer_params)),
        ("tfidf", TfidfTransformer()),
        ("toarray", FunctionTransformer(lambda x: x.toarray())),
    ]
    )

preprocessor = ColumnTransformer(
    transformers=[
        ("cat", categorical_transformer, categorical_features),
        ("txt", text_transformer, textual_feature),
    ]
)

sgd_log_pipeline = Pipeline(
    [
        ("preprocessor", preprocessor),
        ('smote', SMOTENC(random_state=11,categorical_features=[4,5,6,7])),
        ("clf", SGDClassifier()),
    ]
)

Answer 1

由于您使用的是SMOTENC，所以不需要进行一次热编码。您可以签出源代码，并将看到它对您提供的分类特性执行一个热编码。

一种解决方案是对您的分类特性进行序号编码，并让SMOTENC将它们作为分类处理。

使用示例数据集：

from sklearn.preprocessing import OneHotEncoder, FunctionTransformer, OrdinalEncoder
from imblearn.pipeline import Pipeline 
from sklearn.feature_extraction.text import CountVectorizer,TfidfTransformer
from sklearn.compose import ColumnTransformer
from imblearn.over_sampling import SMOTENC
from sklearn.linear_model import SGDClassifier

from sklearn.datasets import fetch_20newsgroups
newsgroups_train = fetch_20newsgroups(subset='train',categories=['rec.autos','sci.space','comp.graphics'])

n = len(newsgroups_train.data)
X = pd.DataFrame(np.random.choice(['A','B','C'],(n,4)),columns = ['F1','F2','F3','F4'])
X['F5'] = newsgroups_train.data

y = (newsgroups_train.target == 2).astype(int)

你的矢量部分：

vectorizer_params = dict(ngram_range=(1, 2), min_df=200, max_df=0.8)

textual_feature = ['F5']
text_transformer = Pipeline(
    steps=[
        ("squeez", FunctionTransformer(lambda x: x.squeeze())),
        ("vect", CountVectorizer(**vectorizer_params)),
        ("tfidf", TfidfTransformer()),
        ("toarray", FunctionTransformer(lambda x: x.toarray())),
    ]
    )

使用序数编码器，而不是单个编码器：

categorical_features = ['F1','F2','F3','F4']
categorical_transformer = OrdinalEncoder()

对于管道的其余部分，以及输入参数categorical_features= ( SMOTENC )，我们将放置前4列，因为您有4个分类特性：

preprocessor = ColumnTransformer(
    transformers=[
        ("cat", categorical_transformer, categorical_features),
        ("txt", text_transformer, textual_feature),
    ]
)

sgd_log_pipeline = Pipeline(
    [
        ("preprocessor", preprocessor),
        ('smote', SMOTENC(random_state=11,categorical_features=[0,1,2,3])),
        ("clf", SGDClassifier()),
    ]
)

因此，我们测试输入的预处理部分，从您的文本向量器中，我们期望输出如下：

text_transformer.fit_transform(X[textual_feature]).shape
(1771, 178)

加上我们的四个序数编码特性，我们期望的是来自预处理程序的输出：

preprocessor.fit_transform(X).shape
(1771, 182) 

#for display purpose
preprocessor.fit_transform(X).round(3)
 
array([[0.   , 0.   , 0.   , ..., 0.308, 0.   , 0.361],
       [1.   , 2.   , 2.   , ..., 0.252, 0.12 , 0.099],
       [2.   , 0.   , 1.   , ..., 0.05 , 0.   , 0.   ],
       ...,
       [1.   , 1.   , 2.   , ..., 0.119, 0.226, 0.   ],
       [1.   , 1.   , 0.   , ..., 0.   , 0.   , 0.   ],
       [1.   , 0.   , 1.   , ..., 0.   , 0.   , 0.   ]])

在本例中，前4列是您的4个分类特性，序号编码。最后的矩阵是4+(文本向量器的特性)。

让我们配合一下，我们可以检查分类器中的内容：

sgd_log_pipeline.fit_resample(X,y)
sgd_log_pipeline.named_steps['clf'].n_features_in_

182