Dataproc自定义映像:无法找到带有dataproc版本的dataproc基映像

Question

我试图创建一个谷歌数据自定义图像并面对这个问题.

$ python generate_custom_image.py --image-name 1.5.48-ubuntu18-custom --dataproc-version 1.5.48-ubuntu18 --customization-script my-customization-script.sh --zone us-central1 --gcs-bucket gs://dataproc-327519-imgs
INFO:__main__:Parsed args: Namespace(accelerator=None, base_image_family=None, base_image_uri=None, customization_script='my-customization-script.sh', dataproc_version='1.5.48-ubuntu18', disk_size=20, dry_run=False, extra_sources={}, family='dataproc-custom-image', gcs_bucket='gs://dataproc-327519-imgs', image_name='1.5.48-ubuntu18-custom', machine_type='n1-standard-1', metadata=None, network='', no_external_ip=False, no_smoke_test=False, oauth=None, project_id=None, service_account='default', shutdown_instance_timer_sec=300, storage_location=None, subnetwork='', zone='us-central1')
INFO:custom_image_utils.args_inferer:Getting Dataproc base image name...
Traceback (most recent call last):
  File "generate_custom_image.py", line 95, in <module>
    main()
  File "generate_custom_image.py", line 86, in main
    args = parse_args(sys.argv[1:])
  File "generate_custom_image.py", line 57, in parse_args
    args_inferer.infer_args(args)
  File "/home/gdataproc/custom-images/custom_image_utils/args_inferer.py", line 225, in infer_args
    _infer_base_image(args)
  File "/home/gdataproc/custom-images/custom_image_utils/args_inferer.py", line 191, in _infer_base_image
    args.dataproc_version)
  File "/home/gdataproc/custom-images/custom_image_utils/args_inferer.py", line 175, in _get_dataproc_image_path_by_version
    "Cannot find dataproc base image with dataproc-version=%s." % version)
RuntimeError: Cannot find dataproc base image with dataproc-version=1.5.48-ubuntu18.

知道为什么吗？

Answer 1

自定义图像脚本依赖于图像上的标签goog-dataproc-version来将图像子版本(例如，1.5.48-ubuntu18)解析为特定的图像URI，但是由于发布过程中的问题，可能会延迟将标签添加到新发布的图像中。这就是为什么有时用户会看到上面的错误。

解决办法：

Dataproc版本页https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-1.5中的

1. 选择较早的子次版本。

1. 或使用--dataproc-version <minor-version> (例如，1.5-ubuntu18)让脚本自动解析到最新可用的子次要版本。您应该能够通过用gcloud compute images describe <custom-image>描述您的自定义图像并检查dataproc-version标签.

来查看它选择的版本。