课堂教学中的师生模式

Question

我正在把url下面的师生模型转换为keras one。

compression/blob/master/teacher-student.py

我如何输入两个模型(学生，老师)，并得到一个输出只有一个学生在角角？我将使用trainable=false设置教师的所有张量，并将损失函数设置为学生输出与教师输出的差，如下所示：

tf_loss = tf.nn.l2_loss(teacher - student)/batch_size

如我所知，在定义model.fit时，只向一个模型提供输入是可能的。但在这种情况下，我应该同时向老师和学生的模式。

提前感谢！

Answer 1

我在Keras中看到的唯一实现是构建两个独立的函数，这些函数可以拓宽或加深教师模型中的权重层，作为学生模型的初始权重。我不确定是否正是Hinton等人。(2015年)老实说，这是升华，但它是师生。https://github.com/fchollet/keras/issues/3491

Answer 2

下面是非常简单的师生模型。我希望这对像我这样的人有帮助。做得好!

import keras
from keras.datasets import mnist
from keras.layers import Input, Embedding, LSTM, Dense, Lambda
from keras.models import Model
import numpy as np
from keras.utils import np_utils
from keras.layers.core import Dense, Dropout, Activation

nb_classes = 10

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)



from keras.models import Sequential
from keras.layers import Dense, Merge
from keras.optimizers import SGD, Adam, RMSprop

batch_size = 128
nb_classes = 10
nb_epoch = 3

teacher = Sequential()
teacher.add(Dense(10, input_shape=(784,)))
teacher.add(Dense(10))
teacher.add(Activation('softmax'))

teacher.summary()
teacher.compile(loss='categorical_crossentropy',
              optimizer=RMSprop(),
              metrics=['accuracy'])

history = teacher.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    verbose=1, validation_data=(X_test, Y_test))
score = teacher.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

for i in range(len(teacher.layers)):
    setattr(teacher.layers[i], 'trainable', False)



Y_train = np.zeros((60000, 10))



student = Sequential()
student.add(Dense(10, input_dim=784))
student.add(Activation('softmax'))
student.compile(loss='mean_squared_error', optimizer='Adam', metrics=['accuracy'])

from keras.layers import *
def negativeActivation(x):
    return -x

negativeRight = Activation(negativeActivation)(student.output) 
diff = Add()([teacher.output,negativeRight])


model = Model(inputs=[teacher.input, student.input], outputs=[diff])
model.compile(loss='mean_squared_error', optimizer='Adam', metrics=['acc'])

model.summary(line_length=150)
model.fit([X_train, X_train], [Y_train], batch_size=128, nb_epoch=5)


print student.evaluate(X_test, Y_test)