从TF-YARN库创建用于分布式训练的pex时出错

Question

我们正在尝试TF-YARN库，用于在tendorflow上训练DL，因为我们的数据在Hadoop中。但是我们在cluster_pack.upload_env()中遇到了错误

以下是完整的错误：

ERROR:cluster_pack.packaging:Cannot创建pex回溯(最近一次调用)：File "/data1/python3.6.10/lib/python3.6/site-packages/cluster_pack/packaging.py"，行144，in pack_in_pex indexes=CRITEO_PYPI_URL if _is_criteo() else None) File pex行803，在返回list(resolve_request.resolve_distributions(ignore_errors=ignore_errors))文件"/data1/python3.6.10/lib/python3.6/site-packages/pex/resolver.py"，行500中，在raise_type=Unsatisfiable中)：文件"/data1/python3.6.10/lib/python3.6/site-packages/pex/resolver.py"，行370，在"/data1/python3.6.10/lib/python3.6/site-packages/pex/jobs.py"，_run_parallel max_jobs=self._max_parallel_jobs文件第219行中，在execute_parallel raise pex.resolver.Unsatisfiable中: pid: 6749 -> /data1/python3.6.10/bin/python3.6 /tmp/tmpizknr9r --disable-pip-version-check --isolated --exists-action I -q --无缓存目录下载--dest /tmp/tmp1ezcnpuj/resolved_dists/cp36-cp36mabsl-py=0.9.0 alembic==1.4.2 astor==0.8.1 astunparse==1.6.3异步生成器==1.10 attrs==19.3.0 backcall==0.1.0 bleach==3.1.5cachetools==4.1.1 certifi==2020.4.5.1 certipy==0.1.3 cffi==1.14.0 chardet==3.0.4 cloudpickle==1.3.0群集-pack==0.0.9 conda-pack==0.4.0 cryptography==2.9.2 cx-Oracle==7.3.0 cycler==0.10.0 decorator==4.4.2 defusedxml==0.6.0 entrypoints==0.3 gast==0.3.3 google-auth==1.18.0 google-auth-oauthlib==0.4.1 google-icc==0.2.0 graphframes==0.6 grpcio==1.30.0 h5py==2.10.0 icc-rt==2020.0.133 idna==2.9 importlib-metadata==1.6.0英特尔-openmp==2020.0.133 ipykernel==5.3.0 ipython==7.14.0 ipython-genutils==0.2.0 ipywidgets==7.5.1 jedi==0.17.0 Jinja2==2.11.2 joblib==0.16.0 json5==0.9.4 jsonschema==3.2.0 jupyter-client==6.1.3 jupyter-core==4.6.3 jupyter-telemetry==0.1.0 jupyter-tensorboard==0.2.0 jupyterhub==1.1.0 jupyterlab==2.1.2 jupyterlab-server==1.1.4 Keras==2.4.3 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 kiwisolver==1.2.0 Mako==1.1.2 Markdown==3.2.2 MarkupSafe==1.1.1 matplotlib==3.2.2 mistune==0.8.4 mkl==2019.0 mkl-random==1.0.1.1 nbconvert==5.6.1 nbformat==5.0.6 networkx==2.4 nose==1.3.7 notebook==6.0.3 numpy==1.18.5 oauthlib==3.1.0 opt-einsum==3.2.1 packaging==20.4 pamela==1.0.0 pandas==1.0.4 pandocfilters==1.4.2.1 pexpect==4.8.0 pickleshare==0.7.5 prometheus-pickleshare==0.7.5==0.7.1提示-工具包==3.0.5 protobuf==3.12.2 ptyprocess==0.6.0 py4j==0.10.7 pyarrow==1.0.0 pyasn1==0.4.8 pyasn1-模块==0.2.8 pycparser==2.20 Pygments==2.6.1 pyOpenSSL==19.1.0 pyparsing==2.4.7 pyrsistent==0.16.0 pyspark==2.4.6 python-dateutil==2.8.1 python-编辑器==1.0.4 python-json-logger==0.1.11 pytz==2020.1 PyYAML==5.3.1 pyzmq==19.0.1 requests==2.23.0 requests-oauthlib==1.3.0 rsa==4.6 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 scipy==1.4.1 -learn==0.23.1 seaborn==0.10.1 Send2Trash==1.5.0 six==1.15.0 skein==0.8.0 sklearn==0.0 SQLAlchemy==1.3.17 tbb==2019.0 tbb4py==2019.0 tensorboard==2.2.2 tensorboard-plugin-wit==1.7.0 tensorflow==2.2.0 tensorflow-estimator==2.2.0 tensorflowonspark==2.2.1 termcolor==1.1.0 terminado==0.8.3testpath==0.4.4 tf-yarn==0.5.1 threadpoolctl==2.1.0 tornado==6.0.4 traitlets==4.3.3 urllib3==1.25.9 wcwidth==0.1.9 webencodings==0.5.1 Werkzeug==1.0.1 widgetsnbextension==3.5.1 wrapt==1.12.1 download引发执行/data1/python3.6.10/bin/python3.6 /tmp/tmpizknr9r --disable-pip-version-check --testpath==0.4.4 -q --存在操作I缓存--无缓存目录下载--dest /tmp/tmp1ezcnpuj/resolved_dists/cp36-cp36mabsl-py==0.9.0 alembic==1.4.2 astor==0.8.1 astunparse==1.6.3 async-generator==1.10 attrs==19.3.0 backcall==0.1.0 bleach==3.1.5 cachetools==4.1.1 certifi==2020.4.5.1 certipy==0.1.3 cffi==1.14.0 chardet==3.0.4 cloudpickle==1.3.0 cluster-pack==0.0.9 conda-pack==0.4.0 cryptography==2.9.2 cx-Oracle==7.3.0 cycler==0.10.0 decorator==4.4.2 defusedxml==0.6.0entrypoints==0.3 gast==0.3.3 google-auth==1.18.0 google-auth-oauthlib==0.4.1 google- graphframes==0.6 ==0.2.0 graphframes==0.6 grpcio==1.30.0 h5py==2.10.0 icc-rt==2020.0.133 idna==2.9输入库-metadata==1.6.0英特尔-openmp==2020.0.133 ipykernel==5.3.0 ipython==7.14.0 ipython-genutils==0.2.0 ipywidgets==7.5.1 jedi==0.17.0 Jinja2==2.11.2 joblib==0.16.0 json5==0.9.4 jsonschema==3.2.0 jupyter-jsonschema==3.2.0==6.1.3 jupyter- jupyterlab==2.1.2 ==4.6.3 jupyter-telemetry==0.1.0 jupyter-tensorboard==0.2.0 jupyterhub==1.1.0 jupyterlab==2.1.2 jupyterlab-.3==1.1.4 kiwisolver==1.2.0 Keras Applications==1.0.8 Keras Preprocessing==1.1.2 kiwisolver==1.2.0 Mako==1.1.2 Markdown==3.2.2 MarkupSafe==1.1.1 matplotlib==3.2.2 mistune==0.8.4 mkl==2019.0 mkl-MarkupSafe==1.1.1==1.0.1.1 nbconvert==5.6.1 nbformat==5.0.6 networkx==2.4 nose==1.3.7 notebook==6.0.3 numpy==1.18.5 oauthlib==3.1.0 opt-einsum==3.2.1 packaging==20.4 pamela==1.0.0 pandas==1.0.4 pandocfilters==1.4.2 parso==0.7.0 pex==2.1.1 pexpect==4.8.0 pickleshare==0.7.5 prometheus-client==0.7.1 prometheus toolkit==3.0.5 protobuf==3.12.2 ptyprocess==0.6.0 py4j==0.10.7 pyarrow==1.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.20 Pygments==2.6.1 pyOpenSSL==19.1.0。4.7 pyrsistent==0.16.0 pyspark==2.4.6 python-dateutil==2.8.1python-编辑器==1.0.4 python-json-logger==0.1.11 pytz==2020.1 PyYAML==5.3.1 pyzmq==19.0.1 requests==2.23.0 requests-oauthlib==1.3.0 rsa==4.6 ruamel.yaml==0.16.10 ruamel.yaml.clib==0.2.0 ruamel.yaml.clib==0.2.0-learn==0.23.1 scipy==1.4.1 seaborn==0.10.1 Send2Trash==1.5.0 six==1.15.0 skein==0.8.0 sklearn==0.0 SQLAlchemy==1.3.17tbb==2019.0 tbb4py==2019.0 tensorboard==2.2.2 tensorboard-plugin-wit==1.7.0 tensorflow==2.2.0 tensorflow-estimator==2.2.0 tensorflowonspark==2.2.1 termcolor==1.1.0 terminado==0.8.3 testpath==0.4.4 tf-yarn==0.5.1 threadpoolctl==2.1.0 tornado==6.0.4 traitlets==4.3.3 urllib3==1.25.9 wcwidth==0.1.9 webencodings==0.5.1 Werkzeug==1.0.1 widgetsnbextension==3.5.1 wrapt==1.12.1 zipp==3.1.0失败，错误为120

Answer 1

使用您的一个依赖项创建pex失败的原因是什么。你真的有很多依赖项。最好的方法是隔离每个用例的依赖项，并创建一个较小的虚拟环境，或者仅使用tensorflow进行尝试。

您可以尝试执行具有不同需求的pex命令，并查看到底是哪个需求造成了问题。同样重要的是要检查它是否能与更新的pex版本一起工作(目前tf-yarn使用pex==2.1.1)。

pex -r requirements -o myarchive.pex

作为替代，你也可以尝试一下，如果它与conda一起工作。

如果您创建了一个具有绝对必要需求的requirements.txt文件，并在https://github.com/criteo/tf-yarn/issues中输入了一个问题，我可以查看一下。