构建一个sklearn文本分类器并使用coremltools进行转换

Question

我想要建立一个文本分类器与学习，然后将它转换为iOS11机器学习文件使用协同工具包。我用Logistic回归、随机林和线性SVC构建了三个不同的分类器，它们都在Python中工作得很好。问题在于coremltools包及其将sklearn模型转换为iOS文件的方式。正如其文件所说，它只支持这些模型：

• 线性和Logistic回归
• LinearSVC和LinearSVR
• SVC和SVR
• NuSVC和NuSVR
• 梯度增强分类器与回归器
• 决策树分类器与回归器
• 随机森林分类与回归
• 正火器
• 进位机
• 标准标尺
• DictVectorizer
• 一个热编码器

因此，它不允许我将文本数据集向量化(我在分类器中使用了TfidfVectorizer包)：

import coremltools
coreml_model = coremltools.converters.sklearn.convert(model, input_features='text', output_feature_names='category')

Traceback (most recent call last):

File "<ipython-input-3-97beddbdad10>", line 1, in <module>
    coreml_model = coremltools.converters.sklearn.convert(pipeline, input_features='Message', output_feature_names='Label')

  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter.py", line 146, in convert
    sk_obj, input_features, output_feature_names, class_labels = None)

  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter_internal.py", line 147, in _convert_sklearn_model
    for sk_obj_name, sk_obj in sk_obj_list]

  File "/usr/local/lib/python2.7/dist-packages/coremltools/converters/sklearn/_converter_internal.py", line 97, in _get_converter_module
    ",".join(k.__name__ for k in _converter_module_list)))

ValueError: Transformer 'TfidfVectorizer(analyzer='word', binary=False, decode_error=u'strict',
        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',
        lowercase=True, max_df=1.0, max_features=None, min_df=3,
        ngram_range=(1, 2), norm=u'l2', preprocessor=None, smooth_idf=1,
        stop_words='english', strip_accents='unicode', sublinear_tf=1,
        token_pattern='\\w+', tokenizer=None, use_idf=1, vocabulary=None)' not supported; 
supported transformers are coremltools.converters.sklearn._dict_vectorizer,coremltools.converters.sklearn._one_hot_encoder,coremltools.converters.sklearn._normalizer,coremltools.converters.sklearn._standard_scaler,coremltools.converters.sklearn._imputer,coremltools.converters.sklearn._NuSVC,coremltools.converters.sklearn._NuSVR,coremltools.converters.sklearn._SVC,coremltools.converters.sklearn._SVR,coremltools.converters.sklearn._linear_regression,coremltools.converters.sklearn._LinearSVC,coremltools.converters.sklearn._LinearSVR,coremltools.converters.sklearn._logistic_regression,coremltools.converters.sklearn._random_forest_classifier,coremltools.converters.sklearn._random_forest_regressor,coremltools.converters.sklearn._decision_tree_classifier,coremltools.converters.sklearn._decision_tree_regressor,coremltools.converters.sklearn._gradient_boosting_classifier,coremltools.converters.sklearn._gradient_boosting_regressor.

是否有任何方法来建立一个学习文本分类器，而不使用TfidfVectorizer或CountVectorizer模型？

Answer 1

现在，如果您想要将TF-下手矢量器转换为.mlmodel格式，则不能在管道中包含它。方法是将数据分别矢量化，然后训练模型(线性SVC，随机森林，.)用矢量化的数据。然后，您需要计算设备上的tf-国防军表示，然后可以插入模型。这是我写的tf-国防军功能的副本。

func tfidf(document: String) -> MLMultiArray{
    let wordsFile = Bundle.main.path(forResource: "words_ordered", ofType: "txt")
    let dataFile = Bundle.main.path(forResource: "data", ofType: "txt")
    do {
        let wordsFileText = try String(contentsOfFile: wordsFile!, encoding: String.Encoding.utf8)
        var wordsData = wordsFileText.components(separatedBy: .newlines)
        let dataFileText = try String(contentsOfFile: dataFile!, encoding: String.Encoding.utf8)
        var data = dataFileText.components(separatedBy: .newlines)
        let wordsInMessage = document.split(separator: " ")
        var vectorized = try MLMultiArray(shape: [NSNumber(integerLiteral: wordsData.count)], dataType: MLMultiArrayDataType.double)
        for i in 0..<wordsData.count{
            let word = wordsData[i]
            if document.contains(word){
                var wordCount = 0
                for substr in wordsInMessage{
                    if substr.elementsEqual(word){
                        wordCount += 1
                    }
                }
                let tf = Double(wordCount) / Double(wordsInMessage.count)
                var docCount = 0
                for line in data{
                    if line.contains(word) {
                        docCount += 1
                    }
                }
                let idf = log(Double(data.count) / Double(docCount))
                vectorized[i] = NSNumber(value: tf * idf)
            } else {
                vectorized[i] = 0.0
            }
        }
        return vectorized
    } catch {
        return MLMultiArray()
    }
}

编辑:在http://gokulswamy.me/imessage-spam-detection/上写了一篇关于如何做到这一点的文章。