Providing a process pool based on environment

[iparask]

make_atomic_filterbin_map uses MPI and multiprocessing pool to launch processes. This can create conflicts and issues in HPC environments where process creation is not dynamic.

This PR provides a function that discovers the execution environment and returns an executor which can either be an mpipoolexecutor or a poolexecutor.

This work depends on simonsobs/pixell#286

[msilvafe]

Under what conditions does this fail? Only when an environment is loaded that doesn't include mpi? If the mpi package is loaded but the environment is setup to use just 1 node (i.e. like how we've been running so far at nersc + tiger) will this also fail and move on to setup the ProcessPoolExecutor?

[iparask]

If an MPI implementation is detected it will use the MPIPoolExecutor and assign as many workers as the MPI world. If an MPI implementation is not detected it will use the ProcessPoolExecutor. For example:

ip8725@tiger-g04c1n6:/scratch/gpfs/SIMONSOBS/users/ip8725/tmp$ python mpi_futures.py
<class 'concurrent.futures.process.ProcessPoolExecutor'>
ip8725@tiger-g04c1n6:/scratch/gpfs/SIMONSOBS/users/ip8725/tmp$ module load soconda/3.10/20250109
Loading soconda/3.10/20250109
  Loading requirement: openmpi/gcc/4.1.6 hdf5/gcc/openmpi-4.1.6/1.14.4 anaconda3/2024.2 fftw/gcc/openmpi-4.1.6/3.3.10
(/scratch/gpfs/SIMONSOBS/env/20250109/soconda_3.10) ip8725@tiger-g04c1n6:/scratch/gpfs/SIMONSOBS/users/ip8725/tmp$ srun -n 2 python mpi_futures.py
<class 'mpi4py.futures.pool.MPIPoolExecutor'>

Tiger3 does not allow python mpi_futures.py while MPI is active (because of openmpi) and I have not found a way to go around it and use ProcessPoolExecutor yet.

I am not sure what will happen if I run python mpi_futures.py on NERSC while soconda is active.

I'm sure there are differences on how NERSC and Princeton configures SLURM.

Everything depends on the software execution environment and not on the number of nodes. That means that if MPI is enabled, even with 1 node, it will use the MPIPoolExecutor.

[mhasself]

I know this is a draft, but the extensive reformatting in make_atomic_filterbin_map.py is not acceptable as part of these changes. Substantive changes and formatting changes that have no effect must be in different PRs (or, if same PR, separate commits that are preserved on merge).

[chervias]

We are talking to @iparask about this. We should probably merge the new mapmaker code from #1044 first and then modify on top of that for this stuff.

[iparask]

@mhasself the formatting is because everything goes under the if rank == 0 statement. The call formatting I will remove.

I will see if I can find a way to avoid if rank == 0 in the while script.