MatConvNet中的Adam优化器

Question

我尝试实现Adam，而不是默认的SGD优化器，方法是在cnn_train中更改以下代码：

opts.solver = [] ;  % Empty array means use the default SGD solver
[opts, varargin] = vl_argparse(opts, varargin) ;
if ~isempty(opts.solver)
  assert(isa(opts.solver, 'function_handle') && nargout(opts.solver) == 2,...
    'Invalid solver; expected a function handle with two outputs.') ;
  % Call without input arguments, to get default options
  opts.solverOpts = opts.solver() ;
end

至：

opts.solver = 'adam';
[opts, varargin] = vl_argparse(opts, varargin) ;
opts.solverOpts = opts.solver() ;

但是，我得到了一个错误：

Insufficient number of outputs from right hand side of equal sign to satisfy assignment.
Error in cnn_train>accumulateGradients (line 508)
params.solver(net.layers{l}.weights{j}, state.solverState{l}{j}, ...

你们中有人尝试过改变默认编译器吗？我还应该在cnn_train中更改什么？

Adam函数的代码：

function [w, state] = adam(w, state, grad, opts, lr)
%ADAM
%   Adam solver for use with CNN_TRAIN and CNN_TRAIN_DAG
%
%   See [Kingma et. al., 2014](http://arxiv.org/abs/1412.6980)
%    |  ([pdf](http://arxiv.org/pdf/1412.6980.pdf)).
%
%   If called without any input argument, returns the default options
%   structure. Otherwise provide all input arguments.
%   
%   W is the vector/matrix/tensor of parameters. It can be single/double
%   precision and can be a `gpuArray`.
%
%   STATE is as defined below and so are supported OPTS.
%
%   GRAD is the gradient of the objective w.r.t W
%
%   LR is the learning rate, referred to as \alpha by Algorithm 1 in 
%   [Kingma et. al., 2014].
%
%   Solver options: (opts.train.solverOpts)
%
%   `beta1`:: 0.9
%      Decay for 1st moment vector. See algorithm 1 in [Kingma et.al. 2014]
%
%   `beta2`:: 0.999
%      Decay for 2nd moment vector
%
%   `eps`:: 1e-8
%      Additive offset when dividing by state.v
%
%   The state is initialized as 0 (number) to start with. The first call to
%   this function will initialize it with the default state consisting of
%
%   `m`:: 0
%      First moment vector
%
%   `v`:: 0
%      Second moment vector
%
%   `t`:: 0
%      Global iteration number across epochs
%
%   This implementation borrowed from torch optim.adam

% Copyright (C) 2016 Aravindh Mahendran.
% All rights reserved.
%
% This file is part of the VLFeat library and is made available under
% the terms of the BSD license (see the COPYING file).

if nargin == 0 % Returns the default solver options
  w = struct('beta1', 0.9, 'beta2', 0.999, 'eps', 1e-8) ;
  return ;
end

if isequal(state, 0) % start off with state = 0 so as to get default state
  state = struct('m', 0, 'v', 0, 't', 0);
end

% update first moment vector `m`
state.m = opts.beta1 * state.m + (1 - opts.beta1) * grad ;

% update second moment vector `v`
state.v = opts.beta2 * state.v + (1 - opts.beta2) * grad.^2 ;

% update the time step
state.t = state.t + 1 ;

% This implicitly corrects for biased estimates of first and second moment
% vectors
lr_t = lr * (((1 - opts.beta2^state.t)^0.5) / (1 - opts.beta1^state.t)) ;

% Update `w`
w = w - lr_t * state.m ./ (state.v.^0.5 + opts.eps) ;

Answer 1

“等号右手侧输出的数量不足，无法满足任务要求。”您的输出数量似乎与cnn_train所需的不匹配。你能展示你的亚当功能吗？

在最新版本的MatConvNet中

    [net.layers{l}.weights{j}, state.solverState{l}{j}] = ...
        params.solver(net.layers{l}.weights{j}, state.solverState{l}{j}, ...
        parDer, params.solverOpts, thisLR) ;

它似乎与你的亚当函数相匹配

你为什么不试试这个：

opts.solver = @adam;

而不是opts.solver = 'adam'；