如何检验2种分类准确性分数的重要性:配对排列检验

Question

我有一个经过训练的分类器，测试了两个相关的多类分类任务。由于分类任务的每一次试验是相关的，这2组预测构成配对数据。我想运行一个配对排列测试，以了解两个预测集在分类精度上是否存在显着性差异。

因此，我的数据包含2个预测类列表，其中每个预测都与同一索引下的其他测试集中的预测相关。

示例：

actual_classes = [1, 3, 6, 1, 22, 1, 11, 12, 9, 2]
predictions1 = [1, 3, 6, 1, 22, 1, 11, 12, 9 10] # 90% acc.
predictions2 = [1, 3, 7, 10, 22, 1, 7, 12, 2, 10] # 50% acc.

H0:分类准确率没有显着性差异。

如何进行配对排列测试，以检验分类精度差异的意义？

Answer 1

我一直在考虑这个问题，我打算贴出一个提议的解决方案，看看是否有人批准或解释我为什么错了。

actual_classes = [1, 3, 6, 1, 22, 1, 11, 12, 9, 2]
predictions1 = [1, 3, 6, 1, 22, 1, 11, 12, 9 10] # 90% acc.
predictions2 = [1, 3, 7, 10, 22, 1, 7, 12, 2, 10] # 50% acc.
paired_predictions = [[1,1], [3,3], [6,7], [1,10], [22,22], [1,1], [11,7], [12,12], [9,2], [10,10]]

actual_test_statistic = predictions1 - predictions2 # 90%-50%=40 # 0.9-0.5=0.4
all_simulations = [] # empty list
for number_of_iterations:
    shuffle(paired_predictions) # only shuffle between pairs, not within
    simulated_predictions1 = paired_predictions[first prediction of each pair]
    simulated_predictions2 = paired_predictions[second prediction of each pair]
    simulated_accuracy1 = proportion of times simulated_predictions1 equals actual_classes
    simulated_accuracy2 = proportion of times simulated_predictions2 equals actual_classes
    all_simulations.append(simulated_accuracy1 - simulated_accuracy2) # Put the simulated difference in the list

p = count(absolute(all_simulations) > absolute(actual_test_statistic ))/number_of_iterations

如果你有任何想法，请在评论中告诉我。或者更好的是，在你自己的答案中提供你自己的修正版本。谢谢!