Support for non-summed history outputs on multiple MPI tasks

[jmstone]

In GitLab by @jfields7 on Nov 15, 2023, 11:06

Though most history outputs are summed, sometimes one needs to track quantities such as maximum density or temperature during a run. Currently AthenaK assumes all history outputs should be reduced with MPI_SUM, so this requires ugly tricks to work properly on multiple MPI tasks, such as the following eyesore in pgen/dyngr_tov.cpp in z4c-matter-rebase:

#if MPI_PARALLEL_ENABLED
  if (global_variable::my_rank == 0) {
    MPI_Reduce(MPI_IN_PLACE, &rho_max, 1, MPI_ATHENA_REAL, MPI_MAX, 0, MPI_COMM_WORLD);
  } else {
    MPI_Reduce(&rho_max, &rho_max, 1, MPI_ATHENA_REAL, MPI_MAX, 0, MPI_COMM_WORLD);
    rho_max = 0.;
  }
#endif

  // store data in hdata array
  pdata->hdata[0] = rho_max;

It may be useful to consider a way to specify what the expected MPI reduction operation is for a history output. A cursory glance through history.cpp suggests it would be fairly simple to have an additional flag in HistoryData (perhaps defaulting to MPI_SUM) to set this for all user data, but any sort of finer-grained control for individual user history variables in an intelligent way would be a somewhat bigger project.