群集上的dask-yarn :无法连接到应用程序

Question

我正在尝试使用dask-yarn在集群上分发Python作业。

我使用以下代码来创建集群：

from dask_yarn import YarnCluster

cluster = YarnCluster(environment='.conda/envs/myconda', worker_vcores=2, worker_memory='4GB', n_workers=4)
client = Client(cluster)
cluster.shutdown()

我会假设你需要更多的信息来确保它与纱线相连。

错误信息如下：

---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-16-9de74c663703> in <module>()
----> 1 cluster = YarnCluster('myconda.tar.gz')

~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in __init__(self, environment, n_workers, worker_vcores, worker_memory, worker_restarts, worker_env, scheduler_vcores, scheduler_memory, deploy_mode, name, queue, tags, user, skein_client)
    293                                    user=user)
    294 
--> 295         self._start_cluster(spec, skein_client)
    296 
    297     @cached_property

~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in _start_cluster(self, spec, skein_client)
    373             app = skein_client.submit_and_connect(spec)
    374             try:
--> 375                 scheduler_address = app.kv.wait('dask.scheduler').decode()
    376                 dashboard_address = app.kv.get('dask.dashboard')
    377                 if dashboard_address is not None:

~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in wait(self, key, return_owner)
    653                 return res
    654 
--> 655             event = event_queue.get()
    656 
    657         return event.result if return_owner else event.result.value

~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in get(self, block, timeout)
    279         if isinstance(out, Exception):
    280             self._exception = out
--> 281             raise out
    282         return out
    283 

ConnectionError: Unable to connect to application
```python

Answer 1

这可能是因为您的应用程序无法启动(因此无法连接)。我建议您查看日志：

$ yarn logs -applicationId <YOUR APPLICATION ID>

从上面的代码看，有一件事很奇怪，那就是指定的conda环境。看起来您传递的是目录的路径，而不是归档环境的路径--您可能需要阅读此处的文档：http://yarn.dask.org/en/latest/environments.html。