多层前馈网络在TensorFlow中训练失败

Question

我从TensorFlow教程开始，使用单层前馈神经网络对mnist数据集中的图像进行分类。这很好，我得到了测试集上的80+百分比。然后，我试图通过在中间添加一个新的层来将其修改为多层网络。在这个修改之后，我训练网络的所有尝试都失败了。在最初的几次迭代中，网络变得更好一些，但随后在11.35%的准确率上停滞不前。

使用1个隐藏层的前20次迭代：

Train set: 0.124, test set: 0.098
Train set: 0.102, test set: 0.098
Train set: 0.112, test set: 0.101
Train set: 0.104, test set: 0.101
Train set: 0.092, test set: 0.101
Train set: 0.128, test set: 0.1135
Train set: 0.12, test set: 0.1135
Train set: 0.114, test set: 0.1135
Train set: 0.108, test set: 0.1135
Train set: 0.1, test set: 0.1135
Train set: 0.114, test set: 0.1135
Train set: 0.11, test set: 0.1135
Train set: 0.122, test set: 0.1135
Train set: 0.102, test set: 0.1135
Train set: 0.12, test set: 0.1135
Train set: 0.106, test set: 0.1135
Train set: 0.102, test set: 0.1135
Train set: 0.116, test set: 0.1135
Train set: 0.11, test set: 0.1135
Train set: 0.124, test set: 0.1135

不管我训练它多长时间，它都卡在这里了。我已经尝试从校正的线性单位更改为softmax，两者产生相同的结果。我尝试将适应度函数更改为e=(y_true-y)^2。结果相同。

不使用隐藏层的前二十次迭代：

Train set: 0.124, test set: 0.098
Train set: 0.374, test set: 0.3841
Train set: 0.532, test set: 0.5148
Train set: 0.7, test set: 0.6469
Train set: 0.746, test set: 0.7732
Train set: 0.786, test set: 0.8
Train set: 0.788, test set: 0.7887
Train set: 0.752, test set: 0.7882
Train set: 0.84, test set: 0.8138
Train set: 0.85, test set: 0.8347
Train set: 0.806, test set: 0.8084
Train set: 0.818, test set: 0.7917
Train set: 0.85, test set: 0.8063
Train set: 0.792, test set: 0.8268
Train set: 0.812, test set: 0.8259
Train set: 0.774, test set: 0.8053
Train set: 0.788, test set: 0.8522
Train set: 0.812, test set: 0.8131
Train set: 0.814, test set: 0.8638
Train set: 0.778, test set: 0.8604

下面是我的代码：

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Parameters
batch_size = 500

# Create the network structure
# ----------------------------

# First layer
x = tf.placeholder(tf.float32, [None, 784])
W_1 = tf.Variable(tf.zeros([784,10]))
b_1 = tf.Variable(tf.zeros([10]))
y_1 = tf.nn.relu(tf.matmul(x,W_1) + b_1)

# Second layer
W_2 = tf.Variable(tf.zeros([10,10]))
b_2 = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(y_1,W_2) + b_2)

# Loss function
y_true = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(y), reduction_indices=[1]))

# Training method
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Train network
# -------------
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch, batch_labels = mnist.train.next_batch(batch_size)
for i in range(20):
    print("Train set: " + str(sess.run(accuracy, feed_dict={x: batch, y_true: batch_labels}))
            + ", test set: " + str(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))) 
    sess.run(train_step, feed_dict={x: batch, y_true: batch_labels})
    batch, batch_labels = mnist.train.next_batch(batch_size)

所以这段代码不起作用，但是如果我从

y = tf.nn.softmax(tf.matmul(y_1,W_2) + b_2)

至

y = tf.nn.softmax(tf.matmul(x,W_1) + b_1)

然后它就起作用了。我错过了什么？

编辑:现在我让它工作了。需要两个更改，首先将权重初始化为随机值而不是零(是的，实际上是权重需要不为零，尽管使用了relu函数，但偏置为零是可以的)。第二件事对我来说很奇怪:如果我从输出层删除softmax函数，而不是手动应用交叉熵的公式，而是使用softmax_cross_entropy_with_logits(y，y_true)函数，那么它就可以工作。据我所知，这应该是相同的..以前我也尝试过平方误差和，但也不起作用。无论如何，下面的代码是有效的。(虽然很难看，但很好用。)对于10k次迭代，它在测试集上获得了93.59%的准确率，所以在任何方面都不是最优的，但比没有隐藏层的要好。在仅仅20次迭代之后，它已经达到了65%。

import numpy as np
import tensorflow as tf

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

# Parameters
batch_size = 500

# Create the network structure
# ----------------------------

# First layer
x = tf.placeholder(tf.float32, [None, 784])
W_1 = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))
b_1 = tf.Variable(tf.truncated_normal([10], stddev=0.1))
y_1 = tf.nn.relu(tf.matmul(x,W_1) + b_1)

# Second layer
W_2 = tf.Variable(tf.truncated_normal([10,10], stddev=0.1))
b_2 = tf.Variable(tf.truncated_normal([10], stddev=0.1))
y = tf.matmul(y_1,W_2) + b_2

# Loss function
y_true = tf.placeholder(tf.float32, [None, 10])
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y,y_true))

# Training method
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Train network
# -------------
sess = tf.Session()
sess.run(tf.initialize_all_variables())
batch, batch_labels = mnist.train.next_batch(batch_size)
for i in range(10000):
    if i % 100 == 0:
        print("Train set: " + str(sess.run(accuracy, feed_dict={x: batch, y_true: batch_labels}))
                + ", test set: " + str(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))) 
    sess.run(train_step, feed_dict={x: batch, y_true: batch_labels})
    batch, batch_labels = mnist.train.next_batch(batch_size)

Answer 1

以下是一些建议：

1-将标准差添加到两个权重变量初始化，而不是使用zeros进行初始化

weight_1 = tf.Variable(tf.truncated_normal([784,10], stddev=0.1))

2-降低学习率，直到精确值显示出变化的行为。

3-使用RELU时，使用略微正值初始化偏置。这个建议可能与您所看到的问题关系不大。

bias_1 = tf.Variable(tf.constant(.05, shape=[10]))