在随机梯度下降中增加费用的目的是什么？

Question

我正试图实现一个基于斯坦福在他们第一次分配给cs224n的脚手架基础上的SGD。实现是用python实现的。该脚手架如下：

def load_saved_params():
'''A helper function that loads previously saved parameters and resets
iteration start.'''
return st, params, state #st = starting iteration

def save_params(iter, params):
'''saves the parameters'''

现在，主要功能(我使用多个哈希符号跟踪感兴趣的语句)

def sgd(f, x0, step, iterations, postprocessing=None, useSaved=False,
    PRINT_EVERY=10):
""" Stochastic Gradient Descent

Implement the stochastic gradient descent method in this function.

Arguments:
f -- the function to optimize, it should take a single
     argument and yield two outputs, a cost and the gradient
     with respect to the arguments
x0 -- the initial point to start SGD from
step -- the step size for SGD
iterations -- total iterations to run SGD for
postprocessing -- postprocessing function for the parameters
                  if necessary. In the case of word2vec we will need to
                  normalize the word vectors to have unit length.
PRINT_EVERY -- specifies how many iterations to output loss

Return:
x -- the parameter value after SGD finishes
"""

# Anneal learning rate every several iterations
ANNEAL_EVERY = 20000

if useSaved:
    start_iter, oldx, state = load_saved_params()
    if start_iter > 0:
        x0 = oldx
        step *= 0.5 ** (start_iter / ANNEAL_EVERY)

    if state:
        random.setstate(state)
else:
    start_iter = 0

x = x0

if not postprocessing:
    postprocessing = lambda x: x

expcost = None ######################################################

for iter in xrange(start_iter + 1, iterations + 1):
    # Don't forget to apply the postprocessing after every iteration!
    # You might want to print the progress every few iterations.

    cost = None

    ### END YOUR CODE

    if iter % PRINT_EVERY == 0:
        if not expcost:
            expcost = cost
        else:
            expcost = .95 * expcost + .05 * cost ########################
        print "iter %d: %f" % (iter, expcost)

    if iter % SAVE_PARAMS_EVERY == 0 and useSaved:
        save_params(iter, x)

    if iter % ANNEAL_EVERY == 0:
        step *= 0.5

return x

为了我的目的，我不使用费用。但是代码中的费用的目的是什么。在什么情况下可以使用呢？为什么它被用于修改成本函数计算的成本？

Answer 1

如果您注意到，expcost只用于打印成本。这只是一种平滑成本函数的方法，因为它可以明显地从一批一批跳到另一批，尽管模型有所改进