Pitest reports a subset of tests failing, when all the tests are identical

[andrewaylett]

Example project: https://github.com/andrewaylett/gradle-plugins/tree/pitest-fails-green

To check out the cut-down test case:

git clone https://github.com/andrewaylett/gradle-plugins
cd gradle-plugins
git switch pitest-fails-green

To run the unit tests outside Pitest:

./gradlew check

To run Pitest:

./gradlew pitest

What you should see

The unit tests pass when run as plain unit tests. There are two tests, the content is identical, with no assertions, so one expects them to pass or fail together.

When run under pitest, the first of the tests to be executed will fail, subsequent test will pass.

What I expected to see

Both tests pass under pitest, the same as outside of pitest

Notes

This is cut down from a larger test suite -- tests that don't interact with ProjectBuilder.build() pass, the first test that calls build() fails, subsequent tests calling build() all pass.

I'm running on an AMD64 Mac, JDK is Temurin 17.

I've tried to work out why the test is failing, but I've not been able to work out what the actual failure is.

[hcoles]

Hi Andrew,

Thanks for the report. I've just taken a very quick look at this.

The test is failing with

17:44:13 PIT >> INFO : MINION : org.gradle.api.GradleException: Could not inject synthetic classes.
<snip lot of stacktrace>
17:44:13 PIT >> INFO : MINION :         at org.gradle.initialization.DefaultLegacyTypesSupport.injectEmptyInterfacesIntoClassLoader(DefaultLegacyTypesSupport.ja
17:44:13 PIT >> INFO : MINION : Caused by: java.lang.IllegalAccessException: module java.base does not open java.lang 

I had to add a catch block to the test to get this, which isn't great. I think some changes are needed to the junit5 plugin to get errors like this to bubble out to the log as you'd expect.

The reason that one test runs green and the other runs red seems to be that ProjectBuilder only attempts to inject the synthetic class the first time it is run. Some internal state somewhere is changed and the call works the 2nd time.

As to why the tests pass without pitest ... they don't. The test class isn't picked up by gradle check (confirmed by added a fail to the test and running check).

Presumably the tests would fail in the same way when run under normal junit if whatever changes required to pick the test class up are made.

[andrewaylett]

Ah, thanks for looking, that's frustrating -- I tried to slim down my test case but slimmed it down too far and trusted that my tests were still fine and Gradle was skipping them because they were up-to-date, sorry.

I've pushed a new commit, along with an extra (failing) test so the results show up in the Actions output. I see three tests run and one fail when running check, but three tests run and two of them fail when running pitest.

The commit is here: https://github.com/andrewaylett/gradle-plugins/tree/21c9bd92225b03ba33254ed4cd5272b640069e29, hopefully I've got it right this time?

Although having said that, from the error you found I wonder if this is another case of #1164?

[hcoles]

Yes, it seems likely that gradle is using the module path to run the tests while pitest is still using classpath.

I think I will finely have to look at modulepath support. I had hoped to be able to put it off until java 8 was out of common usage, but a lot of pitest users still seem to be tied to it.

For the moment you can work around the issue for this project with

  jvmArgs.add("--add-opens=java.base/java.lang=ALL-UNNAMED")

I've raised

pitest/pitest-junit5-plugin#102

To cover the logging issue.

[andrewaylett]

Thanks -- I can confirm that adding jvmArgs mitigates the issue for my non-cut-down project too. And tells me I've got work to do, but I knew that already :).