如何使用PySpark测量Logistic回归的精度和召回率？

Question

我正在通过数据库在PySpark上使用Logistic回归模型，但我无法获得我的精度和召回率。一切运行正常，我可以获得ROC，但没有属性或库用于Precision和Recall

lrModel = LogisticRegression()

predictions = bestModel.transform(testData)

# Instantiate metrics object
results = predictions.select(['probability', 'label'])
results_collect = results.collect()
results_list = [(float(i[0][0]), 1.0-float(i[1])) for i in results_collect]
scoreAndLabels = sc.parallelize(results_list)

metrics = MulticlassMetrics(scoreAndLabels)

# Overall statistics
precision = metrics.precision()
recall = metrics.recall()
f1Score = metrics.fMeasure()
print("Summary Stats")
print("Precision = %s" % precision)
print("Recall = %s" % recall)
print("F1 Score = %s" % f1Score)

>>>Summary Stats
>>>Precision = 0.0
>>>Recall = 0.0
>>>F1 Score = 0.0

Answer 1

我能够创建我自己的函数来这样做。它返回所有内容，甚至更多。我使用的是mllib包中的"MulticlassMetrics()“。由于它是一个多类，因此它会计算每个标签的度量，因此，您必须指定要检索的标签。

### Model Evaluator User Defined Functions
def udfModelEvaluator(dfPredictions, labelColumn='label'):

    colSelect = dfPredictions.select(
      [F.col('prediction').cast(DoubleType())
       ,F.col(labelColumn).cast(DoubleType()).alias('label')])

    metrics = MulticlassMetrics(colSelect.rdd)

    mAccuracy = metrics.accuracy
    mPrecision = metrics.precision(1)
    mRecall = metrics.recall(1)
    mF1 = metrics.fMeasure(1.0, 1.0)

    mMatrix = metrics.confusionMatrix().toArray().astype(int)    

    mTP = metrics.confusionMatrix().toArray()[1][1]
    mTN = metrics.confusionMatrix().toArray()[0][0]
    mFP = metrics.confusionMatrix().toArray()[0][1]
    mFN = metrics.confusionMatrix().toArray()[1][0]

    mResults = [mAccuracy, mPrecision, mRecall, mF1, mMatrix, mTP, mTN, mFP, mFN, "Return [[0]=Accuracy, [1]=Precision, [2]=Recall, [3]=F1, [4]=ConfusionMatrix, [5]=TP, [6]=TN, [7]=FP, [8]=FN]"]

    return mResults

调用函数的步骤：

metricsList = udfModelEvaluator(predictionsData, "label")
metricsList