embedded-kafka scalatest ClassNotFoundException: scala.collection.GenTraversableOnce

Question

对于Scala来说，第一步是用https://github.com/embeddedkafka/embedded-kafka实现https://github.com/embeddedkafka/embedded-kafka，第二个例子位于自述的顶部

import net.manub.embeddedkafka.EmbeddedKafka
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpecLike

class MinimalTest extends AnyWordSpecLike with Matchers {

  "runs with embedded kafka" should {

    "work" in {
      EmbeddedKafka.start()

      1 + 1 shouldBe 2
      // ... code goes here

      EmbeddedKafka.stop()
    }
  }
}

运行此测试时，故障级别比我所熟悉的要低：

MinimalTest:
runs with embedded kafka
*** RUN ABORTED ***
  java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce
  at com.myorganization.api.MinimalTest.$anonfun$new$2(MinimalTest.scala:13)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.wordspec.AnyWordSpecLike$$anon$3.apply(AnyWordSpecLike.scala:1076)
  at org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
  at org.scalatest.TestSuite.withFixture$(TestSuite.scala:195)
  ...
  Cause: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce
  at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
  at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
  at com.myorganization.api.MinimalTest.$anonfun$new$2(MinimalTest.scala:13)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  ...

我怀疑依赖版本不匹配，但无法发现它。以下是我的相关build.gradle内容：

plugins {
    id 'java'
    id 'scala'
}

task spec(dependsOn: ['testClasses'], type: JavaExec) {
    main = 'org.scalatest.tools.Runner'
    args = ['-R', 'build/classes/scala/test', '-o']
    classpath = sourceSets.test.runtimeClasspath
}

dependencies {
    compile 'io.confluent:kafka-streams-avro-serde:5.4.0'
    compile 'io.github.embeddedkafka:embedded-kafka-streams_2.12:2.4.0'
    compile 'io.github.embeddedkafka:embedded-kafka_2.12:2.4.0'
    compile 'org.apache.avro:avro:1.9.1'
    compile 'org.apache.kafka:kafka-clients:2.4.0'
    compile 'org.apache.kafka:kafka-streams:2.4.0'
    compile 'org.apache.kafka:kafka_2.13:2.4.0'
    compile 'org.scala-lang:scala-reflect:2.12.6'
    testCompile 'io.github.embeddedkafka:embedded-kafka-schema-registry_2.12:5.4.0' // match schema registry version
    testCompile 'io.github.embeddedkafka:embedded-kafka-streams_2.13:2.4.0' // match kafka streams version
    testCompile 'io.github.embeddedkafka:embedded-kafka_2.13:2.4.0' // match kafka version
    testCompile 'org.scala-lang:scala-library:2.13.2'
    testCompile 'org.scalatest:scalatest_2.13:3.1.2'
    testImplementation 'junit:junit:4.11'
    testRuntime 'org.pegdown:pegdown:1.4.2'
}

Answer 1

对于Scala来说，对于传递依赖关系来说，Gradle确实有点滑稽--因为它不会自动计算一组连贯的版本。

“缺失”类scala/collection/GenTraversableOnce是scala-库的一部分，可在2.12.x中获得。

因此，您应该能够通过以下方法来解决这个问题：

在运行时类路径上显式声明scala库版本：implementation group: 'org.scala-lang', name: 'scala-library', version: '2.12.6'

• b.检查2.13版本库的所有其他依赖项(如@ShankarShastri所建议的)。似乎嵌入式-卡夫卡确实有一个2.12兼容的构建在这里：https://mvnrepository.com/artifact/io.github.embeddedkafka/embedded-kafka_2.12/2.5.0

一旦您这样做了，假设您的IDE被设置为与您的build.gradle文件同步，您应该能够查看gradle计算出来的依赖项(声明的和传递的)。如果仍然存在问题，请手动查看这些问题，并查看org.scala-lang:scala-library库是否丢失或声明了两次。如果您有多个声明，您可以查看mvncentral.com中每个库的依赖项。

注:步骤(a)的原因是您的“编译”依赖项中有scala伪项，所以我假设您的所有代码(不仅仅是测试代码)都在使用scala。

Answer 2

我可以确认修复是验证依赖关系。或者更严格地说，检查测试依赖项。

对我来说，我升级了我的SpringBoot版本，这改变了我的spring-kafka-test版本，其中实习生包含了Kafka2.13.x，它最终包含了scala。

我在我的项目的构建文件中使用了mvn dependency:tree，并搜索'2.12‘以查找旧的依赖项来自何处。示例依赖树(删除了不相关的依赖项)。注意，org.apache.kafka:kafka_2.11:jar:0.10.0.0包括在my.company.riptide.api:ness-logger:jar:1.0.0中，但是org.springframework.kafka:spring-kafka-test:jar:2.7.9包含一个更新的版本org.apache.kafka:kafka_2.13:jar:2.7.2

我的解决方案是将kafaka_2.11从ness依赖项中排除出来，如下所示：

<dependency>
  <groupId>my.company.riptide.api</groupId>
  <artifactId>ness-logger</artifactId>
  <version>1.0.0</version>
  <exclusions>
    <exclusion>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka_2.11</artifactId>
    </exclusion>
  </exclusions>
</dependency>

截断mvn dependency:tree输出

[INFO] +- my.company.riptide.api:ness-logger:jar:1.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-validation:jar:2.5.7:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:9.0.55:compile
[INFO] |  |  \- org.hibernate.validator:hibernate-validator:jar:6.2.0.Final:compile
[INFO] |  |     \- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
[INFO] |  +- commons-io:commons-io:jar:2.7:compile
[INFO] |  +- my.company.eis:ness-logging-package:jar:4.0.1:compile
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  |  +- org.apache.avro:avro-compiler:jar:1.8.2:compile
[INFO] |  |  |  +- org.apache.velocity:velocity:jar:1.7:compile
[INFO] |  |  |  \- joda-time:joda-time:jar:2.7:compile
[INFO] |  |  +- org.apache.kafka:kafka_2.11:jar:0.10.0.0:compile
[INFO] |  |  |  +- com.101tec:zkclient:jar:0.8:compile
[INFO] |  |  |  \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.4:compile
[INFO] |  |  \- com.netflix.hystrix:hystrix-core:jar:1.5.18:compile
[INFO] |  |     +- com.netflix.archaius:archaius-core:jar:0.4.1:compile
[INFO] |  |     \- io.reactivex:rxjava:jar:1.3.8:compile
[INFO] |  +- org.springframework.boot:spring-boot-loader-tools:jar:2.5.7:compile
[INFO] |  |  \- org.apache.commons:commons-compress:jar:1.21:compile
[INFO] |  \- my.company.riptide.springboot:graceful-shutdown:jar:1.0.2:compile

[INFO] +- org.springframework.kafka:spring-kafka:jar:2.7.9:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.3.13:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.3.13:compile
[INFO] |  +- org.springframework.retry:spring-retry:jar:1.3.1:compile
[INFO] |  |  \- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- org.apache.kafka:kafka-clients:jar:2.7.2:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.4.5-6:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.7.1:compile
[INFO] |  |  \- org.xerial.snappy:snappy-java:jar:1.1.7.7:compile
[INFO] |  \- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:2.2:compile
[INFO] +- io.cucumber:cucumber-spring:jar:7.0.0:test
[INFO] |  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] +- io.cucumber:cucumber-core:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-gherkin:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-gherkin-messages:jar:7.0.0:test
[INFO] |  +- io.cucumber:messages:jar:17.1.1:test
[INFO] |  +- io.cucumber:tag-expressions:jar:4.0.2:test
[INFO] |  +- io.cucumber:cucumber-expressions:jar:13.0.1:test
[INFO] |  +- io.cucumber:datatable:jar:7.0.0:test
[INFO] |  +- io.cucumber:cucumber-plugin:jar:7.0.0:test
[INFO] |  +- io.cucumber:docstring:jar:7.0.0:test
[INFO] |  +- io.cucumber:html-formatter:jar:17.0.0:test
[INFO] |  \- io.cucumber:create-meta:jar:6.0.1:test

[INFO] +- org.springframework.kafka:spring-kafka-test:jar:2.7.9:test
[INFO] |  +- org.apache.kafka:kafka-clients:jar:test:2.7.2:test
[INFO] |  +- org.apache.kafka:kafka-streams:jar:2.7.2:test
[INFO] |  |  +- org.apache.kafka:connect-json:jar:2.7.2:test
[INFO] |  |  |  \- org.apache.kafka:connect-api:jar:2.7.2:test
[INFO] |  |  \- org.rocksdb:rocksdbjni:jar:5.18.4:test
[INFO] |  +- org.apache.kafka:kafka-streams-test-utils:jar:2.7.2:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:2.7.2:test
[INFO] |  |  +- org.apache.kafka:kafka-raft:jar:2.7.2:test
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.13:jar:2.12.5:test
[INFO] |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:jar:2.12.5:test
[INFO] |  |  +- net.sf.jopt-simple:jopt-simple:jar:5.0.4:compile
[INFO] |  |  +- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] |  |  +- org.scala-lang.modules:scala-collection-compat_2.13:jar:2.2.0:test
[INFO] |  |  +- org.scala-lang.modules:scala-java8-compat_2.13:jar:0.9.1:test
[INFO] |  |  +- org.scala-lang:scala-library:jar:2.13.3:compile
[INFO] |  |  +- org.scala-lang:scala-reflect:jar:2.13.3:test
[INFO] |  |  +- com.typesafe.scala-logging:scala-logging_2.13:jar:3.9.2:test
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.5.9:compile
[INFO] |  |  |  +- org.apache.zookeeper:zookeeper-jute:jar:3.5.9:compile
[INFO] |  |  |  +- org.apache.yetus:audience-annotations:jar:0.5.0:compile
[INFO] |  |  |  \- io.netty:netty-transport-native-epoll:jar:4.1.70.Final:compile
[INFO] |  |  \- commons-cli:commons-cli:jar:1.4:test
[INFO] |  +- org.apache.kafka:kafka_2.13:jar:test:2.7.2:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-api:jar:5.7.2:test
[INFO] |     +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |     \- org.junit.platform:junit-platform-commons:jar:1.7.2:test