ClassNotFoundException: no tests found on Orchestrator sharded runs

[frogermcs]

When I'm running sharded instrumentation tests with Orchestrator from the command line I encounter nondeterministic behavior which strongly depends on testing package name (or class name maybe?).
One version works fine, tests from the package are taken one after another. In another version I encounter errors:

java.lang.ClassNotFoundException: Invalid name: no tests found
    at java.lang.Class.classForName(Native Method)
    at java.lang.Class.forName(Class.java:453)
    at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:72)
    at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
    at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:789)
    at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:543)
    at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:386)
    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2145)

It always happens for shardIndex=0 (shardsNum=2 or 3).

It’s not the cache probably. We can reproduce this issue on different machines.

Command

adb -s emulator-5554 shell 'CLASSPATH=$(pm path androidx.test.services) app_process /       androidx.test.services.shellexecutor.ShellMain am instrument -w -r       -e targetInstrumentation com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunner        -e package com.azimo.sendmoney.instrumentation.azimoTestCases.functional.AccountVerification -e numShards 2 -e shardIndex 0       -e clearPackageData true androidx.test.orchestrator/.AndroidTestOrchestrator'

When I change package name by single letter:
From: com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunner
To (“A” at the end): com.azimo.sendmoney.debug1.test/com.azimo.sendmoney.instrumentation.config.AzimoTestRunnerA

Problems don't happen.

Also if I run package in two shards, index=0 and index=1, despite java.lang.ClassNotFoundException error, all tests are run correctly (but they are all run on the emulator running shardIndex=1).

Versions
Test runner: 1.1.1
Espresso: 3.1.1
Orchestrator: 1.1.1 apk
Test services: 1.1.1 apk
Proguard - tried with minify enabled and disabled. Same results.

Our test runner does nothing unusual.

public class AzimoTestRunner extends AndroidJUnitRunner {

    @Override
    public Application newApplication(ClassLoader cl, String className, Context context)
            throws InstantiationException, IllegalAccessException, ClassNotFoundException {
        return super.newApplication(cl, AzimoTestApplication.class.getName(), context);
    }
}

Interesting fact. When I log values from onCreate method bundle, failing run gets:

shardIndex 0
clearPackageData true
shellExecKey cb88bd77f522eeeb
package com.azimo.sendmoney.instrumentation.azimoTestCases.functional.AccountVerification
listTestsForOrchestrator true
numShards 2
orchestratorService OrchestratorService

[stephane-ducornet-tagheuer]

Same problem.

The following command leads to this error "java.lang.ClassNotFoundException: Invalid name: no tests found" :
ANDROID_SERIAL=emulator-5554 ./gradlew :app:connectedIntegDebugAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=com.compagny.tests.SmokeTests -Pandroid.testInstrumentationRunnerArguments.numShards=2 -Pandroid.testInstrumentationRunnerArguments.shardIndex=0

When shardIndex=1, the class is well found and the tests are well executed.

Versions
Test runner: 1.2.0
Espresso: 3.2.0
Orchestrator: 1.2.0
Test services: 1.2.0

[dmeng]

I'm pretty sure I know what's going on.

Looking at line 338 in TestRequestBuilder, when tests are divided into shards, it's sharded by Description#hashCode. But using hashcodes doesn't guarantee that the tests will be distributed evenly.

It's not that hard to make a small example class with two tests that happen to have hashcodes that are either both odd or both even. Running two shards in that scenario causes both tests to be in one shard and the other shard showing the given error.

[juliocbcotta]

@frogermcs , have you solved the issue?
I have the same problem on Orchestrator 1.3.0...