MPI Error in MOOSE

[dewan1988]

Dear moose experts
Currently, I refined my mesh with ICEM CFD and submitted my job for run but unfortunately it is showing the following MPI related error. I don't know how can I solve this error. Before refinement I didn't get any error during running the job. Could you please help me regarding this issue.

Best regards
Dewan

[lindsayad]

Can you try running in serial or is the problem too large? For debugging in general I would recommend looking at https://mooseframework.inl.gov/application_development/debugging.html With a segmentation fault, there's not much of an error message so I have no idea what the problem is

[GiudGiud]

Signal 9 is a sigkill, so it s likely you ran out of memory.
If it is that (and not some MPI error)
Try using less ranks first. Then to be scalable in memory you will need to distribute the mesh (--distributed-mesh on the command line for example)

If it is truly a MPI error, then you will have to look at using a different MPI distribution most likely.
What does mpicc --version and which mpirun return ?

[dewan1988]

Dear GiudGiud and lindsayad

Thank you for your kind reply and suggestions. Currently, I am using 256 cores and 4 nodes for 400 k elements. Earlier before refining the mesh, I have used only 1 node and 64 cores, then it was not showing any mpi errors. Also, it is not possible to use more than 256 nodes at a time in my hpc. As per your suggestion I splited the mesh in 256 chunk and then aadded --distributed-mesh on the command line and ran ran my job but still it is showing the same mpi errors.
The mpicc --version:

and mpirun: 4.0.2

[dewan1988]

Now, it is showing different ranked error.

[GiudGiud]

You are using our conda MPI on your cluster?

This isn't going to work very well. The MPI distribution has to be built specifically for your architecture, notably the network interconnect

You should remove (deactivate) the conda environment and follow HPC install instructions

Create another moose install to keep the old one functional while you create the new one