Support Customized JUnit5 TestEngine that reads test-scenarios from .json files

[jdneo]

Referenced from @emyklebost:

Hello. I've created a custom JUnit5 TestEngine that reads test-scenarios from .json files. I would like to run these tests in vscode using the junit-platform-console-standalone.jar without a build tool. I am able to do this manually using PowerShell, e.g.: java -jar junit-standalone.jar -cp engine.jar -f test/my.test.json

I've added the Extension Pack for Java and the following in settings.json:

{
    "java.project.referencedLibraries": [
        "engine.jar",
        "junit-standalone.jar"
    ],
}

No tests are discovered in the Testing side-bar and nothing happens when clicking Run Tests. Any suggestions?

I've created a branch with my vscode setup and some screenshots as examples: https://github.com/emyklebost/fhir-tdp/tree/vscode-integration

[emyklebost]

Cucumber's JUnit5 Test Engine is also relevant. Adding support for other DiscoverySelectors than Class- and Method-Selectors will allow for tests written in Gherkin and other custom DSLs to be discovered and runnable in vscode.

[YashinaTatiana]

Hello! @jdneo could you please clarify do I understand correctly that UniqueIdSelector is not supported for now?
No tests found for customized JUnit5 TestEngine in vscode.

[jdneo]

Could you provide more details? Like a sample project or sth?

Currently the test discovery is annotation based. Tests annotated by @Testable will be marked as test items.

[YashinaTatiana]

The idea is similar to running .feature files in Cucumber. Our tests are located in a custom folder and are defined in toml files (contain steps, expected result, etc.) rather than in Java classes, so we don't have @Test annotations. The customized JUnit5 TestEngine is used for test execution.

Do I understand correctly that this configuration is not supported currently in Visual Studio code, since test discovery is annotation based? or maybe there are some additional steps/workaround that can be used to run tests with a custom TestEngine that will detect tests with an overridden discover method?

Thanks

[jdneo]

Since current test discovery is annotation based, looks like your case might not be supported now. Have you tried in Eclipse? If Eclipse supports your case, we should be able to support it too.

[YashinaTatiana]

I was able to run tests in IntelliJ IDEA via Run 'All Tests' button on src/../java (and directory marked as test source root). Tried in Eclipse, getting No tests found using Junit5 error

[YashinaTatiana]

Looks like the custom junit5 TestEngine is not recognized by Visual Studio code. Method discover of custom TestEngine is not invoked. Could you please clarify if integration with customized TestEngine is supported? maybe some additional setup is required ?

[jdneo]

Our implementation is similar with Eclipse. So, I'm afraid it's not supported right now.

Is it possible to share a sample project for your request?

[YashinaTatiana]

got it, thank you!