Android Instrumentation测试使用androidx orchestrator局部失败

Question

我正在尝试让我们的测试在androidx orchestrator下运行。

它似乎部分工作，但并不是所有的测试都被找到并运行。

我已经将相关的测试代码提取到我在项目build.gradle上调用的助手方法中。

ext.enableTesting = { Project project ->
    logger.lifecycle "Enable Testing: " + project.displayName

    project.android {
        defaultConfig {
            testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"

            // runs pm clear after each test invocation
            testInstrumentationRunnerArguments clearPackageData: 'true'
        }

        testOptions {
            execution 'ANDROIDX_TEST_ORCHESTRATOR'
            animationsDisabled true

            unitTests {
                includeAndroidResources = true
            }
        }

        useLibrary 'android.test.runner'
        useLibrary 'android.test.base'
        useLibrary 'android.test.mock'
    }

    project.dependencies {
        // Tests - Including NoOp
        testImplementation 'junit:junit:4.12'

        androidTestUtil 'androidx.test:orchestrator:1.2.0'

        // Core library
        androidTestImplementation 'androidx.test:core:1.2.0'

        // AndroidJUnitRunner and JUnit Rules
        androidTestImplementation 'androidx.test:runner:1.2.0'
        androidTestImplementation 'androidx.test:rules:1.2.0'

        // Assertions
        androidTestImplementation 'androidx.test.ext:junit:1.1.1'
        androidTestImplementation 'androidx.test.ext:truth:1.2.0'

        // Mockito
        testImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
        testImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"
        androidTestImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
        androidTestImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"

    }
    // 'androidx.test.ext:truth:1.2.0' pulls in some conflicting
    // versions with other things we need. In particular it has a more
    // recent version of guava which then screws up grpc stuff.
    project.configurations.all {
        resolutionStrategy {
            force 'com.google.errorprone:error_prone_annotations:2.1.2',
                'com.google.code.findbugs:jsr305:3.0.2',
                'com.google.guava:guava:26.0-android',
                'org.checkerframework:checker-compat-qual:2.5.3'
        }
    }
}

我们的18个测试通过了，logcat对这些测试有以下几点：

06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:13:10.135: I/TestRunner(17920): run started: 1 tests
06-14 03:13:10.139: I/TestRunner(17920): started: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.914: I/TestRunner(17920): finished: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.924: I/TestRunner(17920): run finished: 1 tests, 0 failed, 0 ignored

但是，另一个包中的测试失败，出现以下错误：

06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:15:07.676: I/TestRunner(28814): run started: 1 tests
06-14 03:15:07.676: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.677: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.677: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testRunStarted(OrchestratedInstrumentationListener.java:94)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.SynchronizedRunListener.testRunStarted(SynchronizedRunListener.java:35)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$1.notifyListener(RunNotifier.java:91)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier.fireTestRunStarted(RunNotifier.java:88)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
06-14 03:15:07.677: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.677: E/TestRunner(28814):    at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.677: E/TestRunner(28814):    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.677: E/TestRunner(28814): ----- end exception -----
06-14 03:15:07.680: I/TestRunner(28814): started: testMatcherWithMatchFound(app.impl.matcher.MatcherTest)
06-14 03:15:07.681: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.681: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.681: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testStarted(OrchestratedInstrumentationListener.java:114)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:49)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$3.notifyListener(RunNotifier.java:121)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:118)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:42)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:323)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.Suite.runChild(Suite.java:128)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.Suite.runChild(Suite.java:27)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
06-14 03:15:07.681: E/TestRunner(28814):    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.681: E/TestRunner(28814):    at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.681: E/TestRunner(28814):    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.681: E/TestRunner(28814): ----- end exception -----

在此之后，不再运行任何测试，firebase测试实验室将在构建过程中超时。

有什么建议可以让我们所有的测试运行起来吗？

提前感谢！

Answer 1

事实证明，这是编排框架中的一个错误，目前最好的解决方案是干脆不使用编排。我为此创建了一个问题：

https://issuetracker.google.com/issues/135574731