Tensorflow报告错误AUC

Question

我在使用tensorflow库的AUC时遇到了问题。我每批训练我的模型(卷积神经网络)(我不使用验证集)，在每个时代之后，我使用一个独立的测试集来获得我的评估。问题在AUC的评价中。

在每一批中，我计算训练集的AUC/准确度/丢失/精确/召回/F1_得分，然后汇总这些分数的平均值。当我试图做同样的测试集，我再次计算相同的分数。我注意到，除了AUC之外，所有分数都有不同的值。我认为这是不正确的测试的损失函数增加和AUC也增加。问题是，测试的AUC几乎与训练的AUC完全相同(尽管它们的准确性、丢失误差是完全不同的)。

    with tf.name_scope("output"):
        W = tf.Variable(tf.truncated_normal([num_filters_total, num_classes], stddev=0.1), name="W")
        b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="b")
        scores = tf.nn.xw_plus_b(h_drop, W, b, name="scores")
        predictions = tf.argmax(scores, 1, name="predictions")
        l2_loss += tf.nn.l2_loss(W, name="l2_loss")
        l2_loss += tf.nn.l2_loss(b, name="l2_loss")
        tf.summary.histogram("l2", l2_loss)
        tf.summary.histogram("weigths", W)
        tf.summary.histogram("biases", b)

    with tf.name_scope("auc_score"):
        # labelOut = tf.argmax(y_place_holder, 1)
        probability = tf.nn.softmax(scores)
        # auc_scoreTemp = streaming_auc(y_place_holder, probability, curve="PR")
        auc_scoreTemp = tf.metrics.auc(y_place_holder, probability, curve="PR")
        auc_score = tf.reduce_mean(tf.cast(auc_scoreTemp, tf.float32), name="auc_score")
        tf.summary.scalar("auc_score", auc_score)

    with tf.name_scope("accuracy"):
        labelOut = tf.argmax(y_place_holder, 1)
        correct_prediction = tf.equal(predictions, tf.argmax(y_place_holder, 1), name="correct_prediction")
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32), name="accuracy")
        tf.summary.scalar("accuracy", accuracy)


    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())
    for batch in batches:
        x_batch, y_batch = list(zip(*batch))
        _,  accuracy_train, auc_training,  loss_train,  prec_batch, recall_batch, f1_batch \
            = sess.run([train_step,  accuracy, auc_score,  cross_entropy,  precision_mini_batch,
                        recall_mini_batch, f1_score_min_batch], feed_dict={x_place_holder: x_batch,
                                                                           y_place_holder: y_batch,
                                                                           emb_place_holder: vocab_inv_emb_dset,
                                                                           dropout_keep_prob: dropout_rate})
    ...
       for test_batch in test_batches:
                auc_test = None
                x_test_batch, y_test_batch = list(zip(*test_batch))

                accuracy_test, loss_test,  auc_test = sess.run([accuracy, cross_entropy, auc_score],
                       feed_dict={x_place_holder: x_test_batch,
                       y_place_holder: y_test_batch,
                       emb_place_holder: vocab_inv_emb_dset_val,
                       dropout_keep_prob: 1.0})

我还尝试使用streaming_auc，它总是返回1。

编辑

在每个时代结束时，我通过运行以下命令来重置局部变量：

sess.run(tf.local_variables_initializer())

但是第一批输出的结果确实很糟糕。经过第一批测试后，我得到了与训练结果不接近的测试集的正常结果。我不知道这是否是正确的做法，但结果似乎更现实的方式。

Answer 1

所有tf.metrics都返回一个值和一个更新操作(参见这里)。因此，正如所描述的这里，您希望使用更新op积累值，然后计算auc_score以检索累积值，如下所示：

...
auc_score, auc_op = tf.metrics.auc(y_place_holder, probability, curve="PR")
...
for batch in batches:
  sess.run([train_step,  accuracy, auc_op,  cross_entropy,...)
...
py_auc = sess.run(auc)

编辑--展示tf.metrics.auc和tf.metrics.auc的玩具示例

import tensorflow as tf
from tensorflow.contrib import metrics
batch_sz = 100
noise_mag = 0.5
nloop = 10

tf.set_random_seed(0)

batch_x = tf.random_uniform([batch_sz, 1], 0, 2, dtype=tf.int32)
noise = noise_mag * tf.random_normal([batch_sz, 1])
batch_y = tf.sigmoid(tf.to_float(batch_x) + noise)

auc_val, auc_accum = tf.metrics.auc(batch_x, batch_y)

#note: contrib.metrics.streaming_auc reverses labels, predictions
auc_val2, auc_accum2 = metrics.streaming_auc(batch_y, batch_x)   

with tf.Session() as sess:
    sess.run(tf.local_variables_initializer())

    for i in range(nloop):
        _ = sess.run([auc_accum, auc_accum2])

    auc, auc2 = sess.run([auc_val, auc_val2])
    print('Accumulated AUC = ', sess.run(auc_val))  #0.9238014
    print('Accumulated AUC2 = ', sess.run(auc_val)) #0.9238014