流LSTM像素级分类

Question

如何对LSTM网络进行像素级分类？特别是在Tensorflow。

我的直觉告诉我，输出张量(代码中的pred &y )应该是与输入图像具有相同分辨率的二维张量。换句话说，输入图像为200x200，输出分类为200x200。

Udacity课程包括输入图像为28x28的LSTM网络示例。然而，它是一个图像(作为一个整体手写MNIST数据集)分类网络.

我的想法是，我可以用维度[n_classes]替换所有张量，使用[n_input][n_steps] (下面的代码)。然而，它在矩阵乘法中抛出一个错误。

Udacity示例代码部分如下所示：

n_input = 28 # MNIST data input (img shape: 28*28)
n_steps = 28 # timesteps
n_hidden = 128 # hidden layer num of features
n_classes = 10 # MNIST total classes (0-9 digits)

# tf Graph input
x = tf.placeholder("float", [None, n_steps, n_input])
y = tf.placeholder("float", [None, n_classes])

# Define weights
weights = {
    'hidden': tf.Variable(tf.random_normal([n_input, n_hidden])),
    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))
}
biases = {
    'hidden': tf.Variable(tf.random_normal([n_hidden])),
    'out': tf.Variable(tf.random_normal([n_classes]))
}


def RNN(x, weights, biases):

    # Prepare data shape to match `rnn` function requirements
    # Current data input shape: (batch_size, n_steps, n_input)
    # Permuting batch_size and n_steps
    x = tf.transpose(x, [1, 0, 2])
    # Reshaping to (n_steps*batch_size, n_input)
    x = tf.reshape(x, [-1, n_input])
    # Split to get a list of 'n_steps' tensors of shape (batch_size, n_hidden)
    # This input shape is required by `rnn` function
    x = tf.split(0, n_steps, x)

    # Define a lstm cell with tensorflow
    lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
    pdb.set_trace()
    # Get lstm cell output
    outputs, states = rnn.rnn(lstm_cell, x, dtype=tf.float32)

    # Linear activation, using rnn inner loop last output
    return tf.matmul(outputs[-1], weights['out']) + biases['out']

pred = RNN(x, weights, biases)

然后我的代码看起来如下：

n_input = 200 # data data input (img shape: 28*28)
n_steps = 200 # timesteps
n_hidden = 128 # hidden layer num of features
n_classes = 2 # data total classes (0-9 digits)

# tf Graph input
x = tf.placeholder("float", [None, n_input, n_steps])
y = tf.placeholder("float", [None, n_input, n_steps])


# Define weights
weights = {
    'hidden': tf.Variable(tf.random_normal([n_input, n_hidden]), dtype="float32"),
    'out': tf.Variable(tf.random_normal([n_hidden, n_input, n_steps]), dtype="float32")
}
biases = {
    'hidden': tf.Variable(tf.random_normal([n_hidden]), dtype="float32"),
    'out': tf.Variable(tf.random_normal([n_input, n_steps]), dtype="float32")
}


def RNN(x, weights, biases):

    # Prepare data shape to match `rnn` function requirements
    # Current data input shape: (batch_size, n_steps, n_input)
    # Permuting batch_size and n_steps
    pdb.set_trace()
    x = tf.transpose(x, [1, 0, 2])
    # Reshaping to (n_steps*batch_size, n_input)
    x = tf.reshape(x, [-1, n_input])
    # Split to get a list of 'n_steps' tensors of shape (batch_size, n_hidden)
    # This input shape is required by `rnn` function
    x = tf.split(0, n_steps, x)

    # Define a lstm cell with tensorflow
    lstm_cell = rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
    pdb.set_trace()

    # Get lstm cell output
    outputs, states = rnn.rnn(lstm_cell, x, dtype=tf.float32)

    # Linear activation, using rnn inner loop last output
    # return tf.matmul(outputs[-1], weights['out']) + biases['out']
    return tf.batch_matmul(outputs[-1], weights['out']) + biases['out']

pred = RNN(x, weights, biases)

return tf.batch_matmul(outputs[-1], weights['out']) + biases['out']行就是问题所在。因为outputs是二维张量的向量，而weights['out']是三维张量的向量。

我想也许我可以改变outputs的维度，但是这需要深入到RNN对象中(在API中)。

我在这里有什么选择？我能做些整形吗？如果是这样的话，我应该重塑什么，以什么方式？

Answer 1

您不能使用维数为3的形状[n_hidden, n_input, n_step]的矩阵进行矩阵乘法。

您可以做的是输出一个维度[batch_size, n_input * n_step]的向量，然后将其重塑回[batch_size, n_input, n_step]。

weights = {
    'hidden': ... ,
    'out': tf.Variable(tf.random_normal([n_hidden, n_input * n_steps]), dtype="float32")
}
biases = {
    'hidden': ... ,
    'out': tf.Variable(tf.random_normal([n_input * n_steps]), dtype="float32")
}
# ...

pred = RNN(x, weights, biases)
pred = tf.reshape(pred, [-1, n_input, n_steps])

在你的模型上

但是，您在这里所做的是对图像的每一列都使用RNN。您正在尝试获取图像的每个部分(总共200)，并迭代它，这根本不会给出好的结果。

如果您想要处理图像，我建议您查看一下来自TensorFlow的TensorFlow，在那里您可以学习如何在图像上使用卷积，这比RNN有效得多。