Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[OX-1076] JUnit5: test should fail on container level exception (#32)
It has been observed in AMP that JUnit5 tests can fail silently (no test run) when * exception thrown in `@BeforeAll` functions * exception thrown in private member inline initialization * missing dependency in BUCK config The root cause is the `Junit5TestListener` skips test identifiers of type `org.junit.platform.engine.Type.CONTAINER` which results in exceptions not reported when they are not thrown by `@Test` annotated functions. The PR * removes test identifier type check so that container level failures are reported. * adds two junit libs so that junit 5 test can run in the repo. * uses test's LegacyReportingName so that parameterized test name can be displayed properly * example displayName: `[1] 1` * example legacyReportingName: `test3(int)[1]` * increases test count for each test class by 2 because in junit * class level exceptions are reported at class level * dependency exceptions are reported at junit level ### Sample Test Output #### Before fix ``` <?xml version="1.1" encoding="UTF-8" standalone="no"?> <testcase name="com.facebook.buck.mytest.MyTest" runner_capabilities="simple_test_selector" testprotocol="1.0"> <test name="test" success="true" suite="com.facebook.buck.mytest.MyTest" time="17" type="SUCCESS"/> <test name="test2" success="true" suite="com.facebook.buck.mytest.MyTest" time="1" type="SUCCESS"/> </testcase> ``` #### After fix ``` <?xml version="1.1" encoding="UTF-8" standalone="no"?> <testcase name="com.facebook.buck.mytest.MyTest" runner_capabilities="simple_test_selector" testprotocol="1.0"> <test name="test" success="true" suite="com.facebook.buck.mytest.MyTest" time="11" type="SUCCESS"/> <test name="test2" success="true" suite="com.facebook.buck.mytest.MyTest" time="1" type="SUCCESS"/> <test name="com.facebook.buck.mytest.MyTest" success="true" suite="com.facebook.buck.mytest.MyTest" time="2" type="SUCCESS"/> <test name="JUnit Jupiter" success="true" suite="com.facebook.buck.mytest.MyTest" time="5" type="SUCCESS"/> </testcase> ``` ### Test Failure Example 1 BUCK file missing dependency `//third-party/java/opentest4j:opentest4j` #### Before Fix ``` $ ./buck-out/gen/ce9b6f2e/programs/buck.pex test test/com/facebook/buck/mytest: -f MyTest#test Building: finished in 2.3 sec (100%) 21/21 jobs, 5 updated Total time: 2.9 sec Testing: finished in 1.0 sec RESULTS FOR SELECTED TESTS NO TESTS RAN ``` #### After Fix ``` $ ./buck-out/gen/ce9b6f2e/programs/buck.pex test test/com/facebook/buck/mytest: -f MyTest#test FAILURE com.facebook.buck.mytest.MyTest JUnit Jupiter: TestEngine with ID 'junit-jupiter' failed to execute tests Building: finished in 2.1 sec (100%) 21/21 jobs, 5 updated Total time: 2.7 sec Testing: finished in 1.0 sec (0 PASS/1 FAIL) RESULTS FOR SELECTED TESTS FAIL <100ms 0 Passed 0 Skipped 1 Failed com.facebook.buck.mytest.MyTest FAILURE com.facebook.buck.mytest.MyTest JUnit Jupiter: TestEngine with ID 'junit-jupiter' failed to execute tests org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-jupiter' failed to execute tests Caused by: java.lang.NoClassDefFoundError: org/opentest4j/TestAbortedException Caused by: java.lang.ClassNotFoundException: org.opentest4j.TestAbortedException at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594) at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:904) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527) TESTS FAILED: 1 FAILURE Failed target: //test/com/facebook/buck/mytest:mytest FAIL com.facebook.buck.mytest.MyTest ``` ### Test Failure Example 2 ``` @testinstance(TestInstance.Lifecycle.PER_CLASS) public class MyTest { @BeforeAll public void beforeAll() { throw new RuntimeException("before all exception"); } @test public void test() { } } ``` #### Before Fix ``` $ ./buck-out/gen/ce9b6f2e/programs/buck.pex test test/com/facebook/buck/mytest: -f MyTest#test Building: finished in 0.4 sec (100%) 23/23 jobs, 5 updated Total time: 0.5 sec Testing: finished in 0.6 sec RESULTS FOR SELECTED TESTS NO TESTS RAN ``` #### After Fix ``` $ ./buck-out/gen/ce9b6f2e/programs/buck.pex test test/com/facebook/buck/mytest: -f MyTest#test FAILURE com.facebook.buck.mytest.MyTest MyTest: before all exception Building: finished in 2.2 sec (100%) 23/23 jobs, 23 updated Total time: 2.8 sec Testing: finished in 1.1 sec (1 PASS/1 FAIL) RESULTS FOR SELECTED TESTS FAIL <100ms 1 Passed 0 Skipped 1 Failed com.facebook.buck.mytest.MyTest FAILURE com.facebook.buck.mytest.MyTest MyTest: before all exception java.lang.RuntimeException: before all exception at com.facebook.buck.mytest.MyTest.beforeAll(MyTest.java:16) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) TESTS FAILED: 1 FAILURE Failed target: //test/com/facebook/buck/mytest:mytest FAIL com.facebook.buck.mytest.MyTest ```
- Loading branch information