在Google app Engine中部署应用程序时出现内存错误

Question

我正在尝试在Google App Engine中部署flask API，但是遇到了MemoryError。

运行gcloud app deploy时，我通过终端收到以下消息：

Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f857cba3-1bf5-4384-acec-f1ee3ba97c8b status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"26CADD66","errorMessage":"14 May 2020 13:48:07 INFO     Arguments: ['--parser_script=/usr/local/bin/ftl.par', '--src=', '--entrypoint=', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Unparsed arguments: ['--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Executing ['/usr/local/bin/ftl.par', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\nINFO     FTL version python-v0.17.0\nINFO     Beginning FTL build for python\nINFO     FTL arg passed: virtualenv_dir /env\nINFO     FTL arg passed: ttl 168\nINFO     FTL arg passed: python_cmd /opt/python3.7/bin/python3.7\nINFO     FTL arg passed: cache True\nINFO     FTL arg passed: virtualenv_cmd virtualenv\nINFO     FTL arg passed: entrypoint None\nINFO     FTL arg passed: exposed_ports None\nINFO     FTL arg passed: pip_cmd /env/bin/python3.7 -m pip\nINFO     FTL arg passed: tar_base_image_path None\nINFO     FTL arg passed: export_cache_stats False\nINFO     FTL arg passed: builder_output_path \"\"\nINFO     FTL arg passed: destination_path /srv\nINFO     FTL arg passed: sh_c_prefix False\nINFO     FTL arg passed: base us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00\nINFO     FTL arg passed: cache_key_version v0.17.0\nINFO     FTL arg passed: cache_salt \nINFO     FTL arg passed: cache_repository us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d\nINFO     FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m venv /env\nINFO     FTL arg passed: name us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0\nINFO     FTL arg passed: global_cache False\nINFO     FTL arg passed: upload True\nINFO     FTL arg passed: fail_on_error True\nINFO     FTL arg passed: output_path None\nINFO     FTL arg passed: directory /workspace\nINFO     FTL arg passed: additional_directory /.googleconfig\nINFO     FTL arg passed: verbosity DEBUG\nINFO     starting: full build\nINFO     starting: builder initialization\nINFO     Loading Docker credentials for repository 'us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00'\nINFO     Loading Docker credentials for repository 'us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0'\nINFO     builder initialization took 0 seconds\nINFO     starting: build process for FTL image\nINFO     starting: checking_cached_interpreter_layer\nINFO     starting: check python version\nINFO     `python version` full cmd:\n/opt/python3.7/bin/python3.7 --version\nINFO     `python version` stderr:\n\nINFO     check python version took 0 seconds\nDEBUG   .

在检查GCP上的构建详细信息时，我得到了以下信息：

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
  File "/usr/local/bin/ftl.par/__main__.py", line 54, in main
  File "/usr/local/bin/ftl.par/__main__/ftl/python/builder.py", line 123, in Build
  File "/usr/local/bin/ftl.par/__main__/ftl/common/layer_builder.py", line 42, in BuildLayer
  File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_util.py", line 116, in zip_dir_to_layer_sha
MemoryError

我的requirements.txt如下：

scikit-learn==0.20.3
pandas==1.0.3
xgboost==1.0.2
fuzzywuzzy==0.18.0
python-Levenshtein==0.12.0
gensim==3.8.1
nltk==3.3
Flask==1.1.2
flask-cors==3.0.7
requests==2.23.0

我的app.yaml如下：

runtime: python37
instance_class: F4_1G
resources:
  cpu: 3
  memory_gb: 16
  disk_size_gb: 30

我尝试了很多与cpus和内存相关的配置，但都不起作用。

我认为问题在于我正在尝试加载一个3.5 is的word2vec机型：

model = gensim.models.KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary=True)

Answer 1

在App Engine中，app.yaml可以在instance_class和resources之间使用其中一个标记，因为两者都定义了将分配给应用引擎实例的机器类型。

在App Engine Standar中，使用的标签是´instance_class´，如前所述，所有拥有1 GB磁盘的instance class实例都会在尝试上传大于1 GB存储空间的文件时出错。

如果你想指定更多关于将要使用的机器类型的信息，比如内存、磁盘大小等，那么你需要在App Engine Flex中进行部署，在那里你可以指定资源标签。

要在flex上部署，app.yaml需要指定环境是flex，以及本例中实例的自定义配置。

runtime: python
env: flex

runtime_config:
  python_version: 3

resources:
  cpu: 4
  memory_gb: 16
  disk_size_gb: 30

我将CPU从3修改为4，因为此值应为1或偶数。

考虑到App Engine Flex比标准版更昂贵，并且需要更多的时间来部署。

有时这会发送Error Response: [4] Timed out waiting for the app infrastructure to become healthy.错误，预计会抛出，如果遇到此问题，请在一段时间后重试，如果再次发生，请尝试在不同的区域部署应用程序。