DeepAR超参数整定误差

Question

在尝试初始化超参数优化作业时，在DeepAR上对SageMaker进行调优时遇到一个新问题--在调用test:mean_wQuantileLoss时也会发生此错误。我已经升级了sagemaker包，重新启动了我的实例，重新启动了内核(使用juptyer笔记本)，但是问题仍然存在。

ClientError: An error occurred (ValidationException) when calling the 
CreateHyperParameterTuningJob operation: The objective metric type, [Maximize], that you specified for objective metric, [test:RMSE], isn’t valid for the [156387875391.dkr.ecr.us-west-2.amazonaws.com/forecasting-deepar:1] algorithm. Choose a valid objective metric type.

代码：

my_tuner = HyperparameterTuner(estimator=estimator,
                               objective_metric_name="test:RMSE",
                               hyperparameter_ranges=hyperparams,
                               max_jobs=20,
                               max_parallel_jobs=2)

# Start hyperparameter tuning job
my_tuner.fit(inputs=data_channels)

Stack Trace:
ClientError                               Traceback (most recent call last)
<ipython-input-66-9d6d8de89536> in <module>()
      7 
      8 # Start hyperparameter tuning job
----> 9 my_tuner.fit(inputs=data_channels)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py in fit(self, inputs, job_name, include_cls_metadata, **kwargs)
    255 
    256         self._prepare_for_training(job_name=job_name, include_cls_metadata=include_cls_metadata)
--> 257         self.latest_tuning_job = _TuningJob.start_new(self, inputs)
    258 
    259     @classmethod

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py in start_new(cls, tuner, inputs)
    525                                                output_config=(config['output_config']),
    526                                                resource_config=(config['resource_config']),
--> 527                                                stop_condition=(config['stop_condition']), tags=tuner.tags)
    528 
    529         return cls(tuner.sagemaker_session, tuner._current_job_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in tune(self, job_name, strategy, objective_type, objective_metric_name, max_jobs, max_parallel_jobs, parameter_ranges, static_hyperparameters, image, input_mode, metric_definitions, role, input_config, output_config, resource_config, stop_condition, tags)
    348         LOGGER.info('Creating hyperparameter tuning job with name: {}'.format(job_name))
    349         LOGGER.debug('tune request: {}'.format(json.dumps(tune_request, indent=4)))
--> 350         self.sagemaker_client.create_hyper_parameter_tuning_job(**tune_request)
    351 
    352     def stop_tuning_job(self, name):

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    312                     "%s() only accepts keyword arguments." % py_operation_name)
    313             # The "self" in this scope is referring to the BaseClient.
--> 314             return self._make_api_call(operation_name, kwargs)
    315 
    316         _api_call.__name__ = str(py_operation_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    610             error_code = parsed_response.get("Error", {}).get("Code")
    611             error_class = self.exceptions.from_code(error_code)
--> 612             raise error_class(parsed_response, operation_name)
    613         else:
    614             return parsed_response

ClientError: An error occurred (ValidationException) when calling the CreateHyperParameterTuningJob operation: 
The objective metric type, [Maximize], that you specified for objective metric, [test:RMSE], isn’t valid for the [156387875391.dkr.ecr.us-west-2.amazonaws.com/forecasting-deepar:1] algorithm. 
Choose a valid objective metric type.

Answer 1

看起来，您正在尝试通过测试:RMSE只能最小化 SageMaker HyperParameter调优来最大化这个度量。

要在SageMaker Python中实现这一点，请使用objective=‘HyperparameterTuner’创建您的HyperparameterTuner。您可以看到init方法这里的签名。

下面是您对HyperparameterTuner的调用应该做的更改：

my_tuner = HyperparameterTuner(estimator=estimator,
                               objective_metric_name="test:RMSE",
                               objective_type='Minimize',
                               hyperparameter_ranges=hyperparams,
                               max_jobs=20,
                               max_parallel_jobs=2)