Using custom target for rerun jobs

[sophia-guo]

Rerun job with failed targets might be costly. There might be a few test cases failures but the test target might include a few hundred of test cases. Also with failed targets some original passing test cases may fail as intermittent failure or fail on some agents.

For openjdk and jck tests rerun job can be updated to use custom targets to only run failed test cases instead of test targets.

One example of running time difference
#5016 (comment)

If the set of failed test cases is too big as jenkins parameter to trigger the job a check can be added to run the failed target instead.

Part of #5016

[sophia-guo]

Openjdk has two custom targets, jck has four custom targets.

Currently each test build can only trigger one custom target. That is if more than one custom target is needed we need to trigger rerun build more than once correspondingly. Or can **_custom target can be triggered as TESTLIST so multiple custom target can be run? ( need to invesitgate).

The other thing is the limitation of the hppt url parameter length. There may be situations that a set of test cases fail ( due to setting issues or machine issues) so the string CUSTOM_TARGET is too long as the url parameter. For this case we may either divide the CUSTOM_TARGET to small sets and trigger the rerun multiple times or just fall back to failed targets. Falling back to failed targets might make more sense as this situation most likely some test cases setting issue or environment issues.

[smlambert]

I think it is fair to simply handle the simple case (which are frequent), logic:

if ((test case failure list belongs to only one type of custom list) && (failed test cases string is below the URI too long limit)) then { 
    trigger rerun as a custom target 
} else {
    trigger the failed targets to be  rerun // as we currently do
}

There are many cases that match this simple case, and it would help to reduce execution time without being too complicated.

[sophia-guo]

Not sure what is the maximum of Adoptiums jenkins header size. As Jenkins header size can be set/configured we may not need to know exactly the size. In most cases failed test cases should not be too long unless some environment issues or test setting issues, for which case we may just switch back to failed targets. I would suggest set the number of testcases as 50 for now. If not good can change later.

[smlambert]

I would suggest set the number of testcases as 50 for now. If not good can change later.

It would be somewhat easy to test this, by finding one of the machines that throw "no route to host" error and run jdk_net and jdk_nio targets or some such (can look at triage issues to find an example).

[sophia-guo]

jdk11 java/lang/invoke/ on linux-aarch64 has 20+ testcases in ProblemList.txt, see how that goes as the number of testcases for http link length.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10329/ - all passed. might only fail on test-aws-rhel76-armv8-1? adoptium/infrastructure#1118

rerun on test-aws-rhel76-armv8-1 https://ci.adoptium.net/view/Test_grinder/job/Grinder/10330/ - passed.
rerun for jdk17,21,22,23 https://ci.adoptium.net/view/Test_grinder/job/Grinder/10331/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10332/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10333/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10334/

all passed except java/lang/invoke/LFCaching/LFGarbageCollectedTest.java, which is not in the current problem list.

[sophia-guo]

try aix jdk17 java/net, java/nio
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10342/

[sophia-guo]

Manually edit https://ci.adoptium.net/job/Grinder/parambuild/?SDK_RESOURCE=nightly&TARGET=jdk_custom&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=false&ACTIVE_NODE_TIMEOUT=&VENDOR_TEST_DIRS=&EXTRA_DOCKER_ARGS=-v+%24%7BTEST_JDK_HOME%7D%3A%2Fopt%2Fjava%2Fopenjdk&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&OPENJ9_SYSTEMTEST_OWNER_BRANCH=eclipse-openj9%3Amaster&PLATFORM=ppc64_aix&GENERATE_JOBS=false&KEEP_REPORTDIR=true&PERSONAL_BUILD=true&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2Fadoptium%2Faqa-tests.git&RERUN_ITERATIONS=0&LABEL=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&NON_AQA_TEST_REPOS=&ARTIFACTORY_SERVER=&KEEP_WORKSPACE=false&USER_CREDENTIALS_ID=&JDK_VERSION=17&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&UPSTREAM_JOB_NAME=&CLOUD_PROVIDER=&PLATFORM_AND_MACHINE=&CUSTOM_TARGET=java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java+java%2Fnet%2FMulticastSocket%2FSetLoopbackMode.java+java%2Fnet%2FMulticastSocket%2FSetOutgoingIf.java+java%2Fnet%2FMulticastSocket%2FTest.java+java%2Fnet%2FSocketPermission%2FSocketPermissionTest.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FAfterDisconnect.java+java%2Fnio%2Fchannels%2FDatagramChannel%2FSendReceiveMaxSize.java+java%2Fnio%2Fchannels%2FAsyncCloseAndInterrupt.java+sun%2Fmanagement%2Fjdp%2FJdpDefaultsTest.java+sun%2Fmanagement%2Fjdp%2FJdpJmxRemoteDynamicPortTest.java+sun%2Fmanagement%2Fjdp%2FJdpOffTest.java+sun%2Fmanagement%2Fjdp%2FJdpSpecificAddressTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverridePropertyInfoTest.java+java%2Fbeans%2FIntrospector%2F8132566%2FOverrideUserDefPropertyInfoTest.java+java%2Fnet%2FDatagramSocket%2FSendReceiveMaxSize.java+java%2Fnet%2FInet4Address%2FPingThis.java+java%2Fnet%2FMulticastSocket%2FB6427403.java+java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java+java%2Fnet%2FMulticastSocket%2FPromiscuous.java&VENDOR_TEST_SHAS=&JDK_BRANCH=&TEST_IMAGES_REQUIRED=true&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=&JDK_IMPL=hotspot&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&DYNAMIC_COMPILE=false&ADOPTOPENJDK_SYSTEMTEST_OWNER_BRANCH=adoptium%3Amaster&APPLICATION_OPTIONS=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&ARCHIVE_TEST_RESULTS=false&OPENJDK_SHA=&NUM_MACHINES=&BUILD_LIST=openjdk&USE_TESTENV_PROPERTIES=false&UPSTREAM_JOB_NUMBER=&STF_OWNER_BRANCH=adoptium%3Amaster&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None
works ( testcases 97) add one more testcases java%2Fnet%2FMulticastSocket%2FNoLoopbackPackets.java will exceed the maximum length. Considered some test case length may be longer and we really don't need to be that accurate as the failure testcases number is rarely large in normal case. Probably the test case number 80 is big enough.