星星之火结构流卡夫卡版本2

Question

我们把卡夫卡版本从0.9升级到2.0

需要帮助查找火花结构化流的正确客户端库

"org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.3.0"

不管用。

这是它抛出的错误：

11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Completed connection to node -1. Fetching API versions. 11:46:18.061 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Initiating API versions fetch from node -1. 11:46:18.452 [stream execution thread for [id = e393ea37-8009-4ce0-b996-94f767994fb8, runId = bc15eb7d-876d-4e01-8ee5-22205ec7fdcb]] DEBUG org.apache.kafka.common.network.Selector - [Consumer clientId=consumer-2, groupId=spark-kafka-source-8ce7f26f-e342-4b0d-85f1-a9f641b79629-1052905425-driver-0] Connection with kafka-muhammad-45e0.aivencloud.com/18.203.67.147 disconnected java.io.EOFException: null at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:119) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:335) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:296) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:560)

Answer 1

请提供更多信息(错误如何？！)。Kafka经纪商被设计成向后兼容，这有助于那些明显落后于最新API(如Spark)的项目。

您所引用的artifactId spark-streaming-kafka-0-10是用于火花流的，为了使用火花结构流，您需要使用spark-sql-kafka-0-10_2.11