androidx.test:orchestrator in test-only module ('com.android.test')

[asavill]

Description

We are struggling to use the test orchestrator in our new test only module. We had it working fine when the tests were located in the app module.

One observation is that when working within a test-only module, dependencies are brought in slightly differently

Test only

implementation "androidx.test:runner:$androidTestRunnerVersion"
implementation "androidx.test:rules:$androidTestRulesVersion"
implementation "androidx.test.ext:truth:$androidTestTruthExtensionsVersion"

Normal app module

androidTestimplementation "androidx.test:runner:$androidTestRunnerVersion"
androidTestimplementation "androidx.test:rules:$androidTestRulesVersion"
androidTestimplementation "androidx.test.ext:truth:$androidTestTruthExtensionsVersion"

As you can see, the androidTest element is omitted. Test orchestrator is brought in using androidTestUtil.. is there an equivalent syntax for test-only modules? Is it possible that a test-only module ignores the androidTestUtil option?

androidTestUtil 'androidx.test:orchestrator:1.3.0'

Steps to Reproduce

Create a test-only module using 'com.android.test' and attempt to use test orchestrator.

Expected Results

Test orchestrator works as normal

Actual Results

Test orchestrator isn't leveraged.

AndroidX Test and Android OS Versions

'androidx.test:orchestrator:1.3.0' - Android 10/11/9

Link to a public git repo demonstrating the problem:

https://github.com/asavill/test-orchestrator-in-test-only-module-issue

[brettchabot]

Can you post a sample project that illustrates this problem? Either as a zip or in github.

[asavill]

@brettchabot Of course. Provided here https://github.com/asavill/test-orchestrator-in-test-only-module-issue

It's the same test class in both the app module and the test-only module. If you run them and compare you will see the orchestrator is not utilised in the test-only module.

[asavill]

Any update on this?

[bilgehankalkan]

@asavill did you find any solution or workaround for this?

[asavill]

@bilgehankalkan I did not, unfortunately. I retry it every time we update the android gradle plugin but it's still not supported. @brettchabot Do you have any idea if this is known about or on a backlog somewhere? I understand it's probably not highest priority.

[brettchabot]

@yuuki3655 any ideas?

[yuuki3655]

@asavill Did you run it from Android Studio or the command line? I downloaded your example project and ran the command, ./gradlew :app:testOnly:connectedDebugAndroidTest --info, but the test passed and the orchestrator was used as expected.

> Task :app:testOnly:connectedDebugAndroidTest
...
[RemoteAndroidTest]: Running CLASSPATH=$(pm path androidx.test.services) app_process / androidx.test.services.shellexecutor.ShellMain am instrument -r -w -e targetInstrumentation com.example.testonlyorchestrator.test/androidx.test.runner.AndroidJUnitRunner   -e additionalTestOutputDir /storage/emulated/0/Android/data/com.example.testonlyorchestrator/files/test_data -e clearPackageData true androidx.test.orchestrator/androidx.test.orchestrator.AndroidTestOrchestrator on Pixel_3a_XL_API_29(AVD) - 10
Starting 3 tests on Pixel_3a_XL_API_29(AVD) - 10

com.example.testonly.ExampleInstrumentedTest > checkForHelloAndroidTestTeam[Pixel_3a_XL_API_29(AVD) - 10] SUCCESS 

com.example.testonly.ExampleInstrumentedTest > checkForHelloWorld[Pixel_3a_XL_API_29(AVD) - 10] SUCCESS 

com.example.testonly.ExampleInstrumentedTest > useAppContext[Pixel_3a_XL_API_29(AVD) - 10] SUCCESS 

[bilgehankalkan]

@yuuki3655 I also have success execution with orchestrator while running with connectedAndroidTest but this task seems like running all tests at once and there are no visual IDE support.
It's OK for running all tests at once and generated test reports are good for CI builds. But while creating new test methods and classes, I would like to see AS support like any other androidTests.
Is there any way to specify single test method or class while using this command? Maybe I'll create IDE configuration for this task and only specify tests that I just created/modified.

✋: Passing argument -Pandroid.testInstrumentationRunnerArguments.package=package or -Pandroid.testInstrumentationRunnerArguments.class=package.classname while running connectedAndroidTest is working fine.

[yuuki3655]

@bilgehankalkan Yes, we are aware of the feature parity issue between Android Studio and Android Gradle plugin. As an interim solution, please use those instrumentation runner arguments. We are actively working on unifying the test runners in Android Studio and Android Gradle plugin and you can opt-in to it in the latest canary build. Here's the related talk at this year's Google IO: https://www.youtube.com/watch?v=juEkViDyzF8

[robpridham-bbc]

Hi @yuuki3655 - could we have an update on this please? What are your expectations around the current level of support in Bumblebee?

I'm still seeing no orchestrator used, and this error message in the run log:

"Run Android instrumented tests using Gradle" option was ignored because this module type is not supported yet."

but it is possible this is us lagging behind in Gradle version etc.

[yuuki3655]

@robpridham-bbc Since Android Studio BumbleBee, it delegates test execution to AGP. So, when you specify orchestrator in your Gradle build file, AS should also use it as long as your project's AGP version supports it. What AGP version is your project using? Could you try upgrading AGP version to the latest, 7.3.0-alpha07?

[ishaileshmishra]

Hi all, I am facing an issue while running instrumentation est cases in android. Any help appreciated

"Run Android instrumented tests using Gradle" option was ignored because this module type is not supported yet.
Running tests

$ adb shell am instrument -w -m -e package com.contentstack.sdk -e debug false com.example.sdk.test/androidx.test.runner.AndroidJUnitRunner
Timed out waiting for process (com.example.sdk.test) to appear on samsung-sm_g975f-adb-RZ8N31V5ESZ-BFkZv4._adb-tls-connect._tcp..

Thanks in advance

[yuuki3655]

"Run Android instrumented tests using Gradle" option was ignored because this module type is not supported yet.

Does your project has a dynamic module? If so, we're still working on it to support it. Test-only module is supported already. What version of Android Gradle plugin does your project use?

Timed out waiting for process (com.example.sdk.test) to appear on samsung-sm_g975f-adb-RZ8N31V5ESZ-BFkZv4._adb-tls-connect._tcp..

This sounds like a test process has crashed for some reasons. You might get more information by adb logcat.

[wiolettakurekpapaya]

Hello :D is there any progress on this issue?

[yuuki3655]

Hi, this should be fully supported including dynamic module in the latest version of Android Studio (Giraffe). Do you still see the issue?