Activate report grid memory consumption

[ykempf]

Just for laughs? How long has this not been used? Not like I just spent a number of weeks looking into memory consumption...........

[ykempf]

Suggestion from Markus: implement a similar memory report functionality for fsgrid. Not today.

[ykempf]

Example output:

(MEM) tstep 0 t 0 Resident per node (avg, min, max): 2.1022 2.1022 2.1022
(MEM) tstep 0 t 0 High water mark per node                 (GiB) avg: 2.1022 min: 2.1022 max: 2.1022 sum (TiB): 0.00205293 on 1 nodes
(MEM) Total size: 1.46364e+08
(MEM) Total capacity 1.58685e+08
(MEM)   Average capacity: 9.91781e+06 local cells 1.8856e+06 remote cells 8.0322e+06
(MEM)   Max capacity:     1.25012e+07 on  process 6
(MEM)   Min capacity:     7.54602e+06 on  process 2

So let's discuss if we want more info (value and rank of local vs. remote cells' min and max capacity?), and in what format/ordering.

And also, this should list explicitly that we're talking of the spatial cells here, especially if/when we extend this to also report fsgrid memory usage.

[ykempf]

So I was thinking (dangerous, I know) – instead of reducing the capacities per node, wouldn't it be more useful in terms of debugging/troubleshooting to instead report the RES and HWM values per rank as well as per node? Or should this just be all PAPI and capacity reports both per rank and per node, to be complete?

Also, is the consensus that I move the whole shebang to memoryallocation.cpp instead of grid.cpp? I just picked one, I had no particular reason beyond putting all in one spot.

[markusbattarbee]

OK, looks good. I see that Flowthrough_damr is still too sensitive - the grid on the last timestep does not match.
I also see that that does not trigger a failure! I guess any output to STDERR should flag the run as failed...

[ursg]

I like the functionality, and I also like the neighborhood-id-refactoring a lot. I'm just confused because I didn't expect this amount of changes given the PR title.
But: nothing I don't like, so let's merge it!