Orphaned processes cause unusable Windows build node

[nickbattam-tessella]

MPI test failures on the Windows build node left MATLAB processes (MPI worker nodes) running on the machine.

This prevented all subsequent jobs executed against that MATLAB release from starting with a series of Jenkins IO exceptions Unable to delete <job directory and subfolders>.

Build logs from the failing and subsequent aborted PR build are attached to this ticked.

Ed: this is NOT limited to runs with failing/timed out tests; an example has been seen where ALL tests passed and the build succeeded in all aspects except the post-build cleanup.

Output:

• identification of method by which orphaned processes are created
• method to cleanup orphaned MATLAB processes on test failure

[nickbattam-tessella]

aborted-build.log
failing-build.log

[nickbattam-tessella]

Orphaned R2019/smpd process locked _test/test_mpi_wrappers after run with all tests passing.
<jenkins-url>/jenkins/job/PACE-neutrons/job/Herbert/job/Windows-10-2019b/62/

[nickbattam-tessella]

Same problem on job/PACE-neutrons/job/Horace/job/Windows-10-2019b/108
console-108.log