支持向量机性能与AUC评分不一致

Question

我有一个包含病人信息的数据集。它包括多个变量和他们的临床状况(0如果他们是健康的，1如果他们是生病的)。我尝试实现基于这些变量的支持向量机模型来预测病人的状态。

library(e1071)

Index <- 
  order(Ytrain, decreasing = FALSE)

SVMfit_Var <- 
  svm(Xtrain[Index, ], Ytrain[Index],
      type = "C-classification", gamma = 0.005, probability = TRUE, cost = 0.001, epsilon = 0.1)


preds1 <- 
  predict(SVMfit_Var, Xtest, probability = TRUE)
preds1 <- 
  attr(preds1, "probabilities")[,1]

samples <- !is.na(Ytest)
  pred <- prediction(preds1[samples],Ytest[samples])
  AUC<-performance(pred,"auc")@y.values[[1]]


prediction <- predict(SVMfit_Var, Xtest)
xtab <- table(Ytest, prediction)

为了测试模型的性能，我计算了ROC模型，并使用验证集获得了一个AUC = 0.997。但是当我看到这些预测的时候，所有的病人都被认为是健康的。

AUC = 0.997
> xtab
     prediction
Ytest  0  1
    0 72  0
    1 52  0

有人能帮我解决这个问题吗？

Answer 1

你看过概率和拟合值吗？You can read about how works with SVM here。

如果您想查看性能，可以使用库DescTools和函数Conf，也可以使用库caret和函数confusionMatrix。(它们提供相同的输出。)

library(DescTools)
library(caret)

# for the training performance with DescTools
Conf(table(SVMfit_Var$fitted, Ytrain[Index])) 
       # svm.model$fitted, y-values for training

# training performance with caret
confusionMatrix(SVMfit_Var$fitted, as.factor(Ytrain[Index])) 
             # svm.model$fitted, y-values 
                       # if y.values aren't factors, use as.factor()

# for testing performance with DescTools
    # with `table()` in your question, you must flip the order:
         # predicted first, then actual values
Conf(table(prediction, Ytest))

# and for caret
confusionMatrix(prediction, as.factor(Ytest))

您的问题是不可重复的，所以我用iris数据进行了讨论。每次观察的概率都是一样的。我包括了这个，所以你可以用另一个数据集看到这个。

library(e1071)
library(ROCR)
library(caret)

data("iris")

# make it binary
df1 <- iris %>% filter(Species != "setosa") %>% droplevels()
# check the subset
summary(df1)

set.seed(395) # keep the sample repeatable
tr <- sample(1:nrow(df1), size = 70, # 70%
             replace = F)

# create the model
svm.fit <- svm(df1[tr, -5], df1[tr, ]$Species,
               type = "C-classification",
               gamma = .005, probability = T,
               cost = .001, epsilon = .1)

# look at probabilities
pb.fit <- predict(svm.fit, df1[-tr, -5], probability = T) 
            # this shows EVERY row has the same outcome probability distro
pb.fit <- attr(pb.fit, "probabilities")[,1]

# look at performance 
performance(prediction(pb.fit, df1[-tr, ]$Species), "auc")@y.values[[1]]
# [1] 0.03555556  that's abysmal!! 

# test the model
p.fit = predict(svm.fit, df1[-tr, -5])
confusionMatrix(p.fit, df1[-tr, ]$Species)
# 93% accuracy with NIR at 50%... the AUC score was not useful

# check the trained model performance
confusionMatrix(svm.fit$fitted, df1[tr, ]$Species)
# 87%, with NIR at 50%... that's really good