Share multiprocessing pool across ensemble members

[jfrost-mo]

This avoids reinitialising the multiprocessing worker processes for each ensemble member, which is very expensive when using the "spawn" start method. This provides a considerable sppedup for large ensembles.

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

• Documentation has been updated to reflect change.
• New code has tests, and affected old tests have been updated.
• All tests and CI checks pass.
• Ensured the pull request title is descriptive.
• Conda lock files have been updated if dependencies have changed.
• Attributed any Generative AI, such as GitHub Copilot, used in this PR.
• Marked the PR as ready to review.

[github-actions]

[jfrost-mo]

Still not fast, and basically eats the testing speedup that #1071 gave us, but about 10 times faster now.

[jfrost-mo]

To explain, the fork start method, which was being used before, makes a copy of the current process and runs the multiprocessing tasks inside that. Linux uses copy-on-write memory for processes, meaning things are only copied once changed, and as most things are not changed (think imported libraries, etc.) this is very fast.

What the spawn method does is it starts a completely fresh python process that has to first import all of the various libraries before it can do any useful work. This is a lot slower, especially for an interpreted language like python.

Unfortunately using fork can cause issues when your program has multiple threads running inside it. This can happen explicitly (we use threads for speeding up downloading the input data), or incidently, such as a library using threads (I think numpy can, for example).

[jwarner8]

Happy with changes, I think impact won't be too large considering nature of data being used with this diagnostic.