使用PySpark内核时朱庇特笔记本错误:代码失败是因为一个致命错误:发送http请求时出错

Question

我和使用jupyter笔记本电脑的PySpark内核，我已经成功地选择了PySpark内核，但我一直收到以下错误

代码失败是因为一个致命错误:发送http请求和遇到的最大重试错误。有些事情需要尝试：

( a)确保星火有足够的可用资源让木星创建星火环境。

( b)与木星管理员联系，确保火花魔法库配置正确。

c)重新启动内核。

这也是日志

2019-10-10 13:37:43,741 DEBUG   SparkMagics Initialized spark magics.
2019-10-10 13:37:43,742 INFO    EventsHandler   InstanceId: 32a21583-6879-4ad5-88bf-e07af0b09387,EventName: notebookLoaded,Timestamp: 2019-10-10 10:37:43.742475
2019-10-10 13:37:43,744 DEBUG   python_jupyter_kernel   Loaded magics.
2019-10-10 13:37:43,744 DEBUG   python_jupyter_kernel   Changed language.
2019-10-10 13:37:44,356 DEBUG   python_jupyter_kernel   Registered auto viz.
2019-10-10 13:37:45,440 INFO    EventsHandler   InstanceId: 32a21583-6879-4ad5-88bf-e07af0b09387,EventName: notebookSessionCreationStart,Timestamp: 2019-10-10 10:37:45.440323,SessionGuid: d230b1f3-6bb1-4a66-bde1-7a73a14d7939,LivyKind: pyspark
2019-10-10 13:37:49,591 ERROR   ReliableHttpClient  Request to 'http://localhost:8998/sessions' failed with 'HTTPConnectionPool(host='localhost', port=8998): Max retries exceeded with url: /sessions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000013184159808>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))'
2019-10-10 13:37:49,591 INFO    EventsHandler   InstanceId: 32a21583-6879-4ad5-88bf-e07af0b09387,EventName: notebookSessionCreationEnd,Timestamp: 2019-10-10 10:37:49.591650,SessionGuid: d230b1f3-6bb1-4a66-bde1-7a73a14d7939,LivyKind: pyspark,SessionId: -1,Status: not_started,Success: False,ExceptionType: HttpClientException,ExceptionMessage: Error sending http request and maximum retry encountered.
2019-10-10 13:37:49,591 ERROR   SparkMagics Error creating session: Error sending http request and maximum retry encountered.

请注意，我试图在windows上配置它。非常感谢

Answer 1

我也面临同样的问题，您可以不使用PySpark内核(笔记本)而是使用Python3内核(笔记本)来解决这个问题。我使用了以下代码来设置星盘集群：

import pyspark # only run after findspark.init()
from pyspark.sql import SparkSession
# May take awhile locally
spark = SparkSession.builder.appName("test").getOrCreate()
spark

Answer 2

如果您试图通过Livy (例如AWS Glue Development Endpoint)将您的木星笔记本连接到Spark服务器(例如AWS Glue Development Endpoint)，则必须将"localhost“替换为Spark服务器IP地址：

如前所述：https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/

Answer 3

张贴在下面的答案，因为它可能会帮助面对这个问题的人时，使用sagemaker笔记本与Glue Dev端点。

我在我的PySpark内核笔记本中也收到了同样的错误消息。在我的例子中，问题是缺少了附加到笔记本实例的生命周期配置，该配置被某种方式删除。我每天删除并重新创建dev端点，但是它的生命周期配置通常仍然附着在笔记本上。