INSFV Memory Usage

[makeclean]

Hi All

Quick one, a colleague pointed out that he was seeing high memory usage in INSFV, so I took @ChrocheMisawa 's problem as posted in #17760 and cracked up the refinement. Im seeing huge memory increases for memory usage

Suggesting 1e6 elements being 200 Gb of memory! That doesnt seem right, thats huge a post on the OpenFOAM forum (apples and elephants comparison) suggests 1e6 elements ~ 1 Gb of memory, so even accounting for the AD cost, this seems expensive.

Thanks

Andy

[permcody]

Are you running these tests with ReplicatedMesh or DistributedMesh? Note that the former keeps a full copy of the mesh on every processor rank when running in parallel. You can imagine how this can get pretty expensive. For larger problems spread across more ranks, you should be using DistributedMesh. We also have the ability to pre-split the mesh to avoid startup spikes in both terms of memory and time. We can run tens of millions of elements using just hundreds of MBs on each rank in this fashion.

[Mesh]
    ...
    parallel_type = distributed
[]

Here is some information on presplitting:
https://mooseframework.inl.gov/syntax/Mesh/splitting.html

[makeclean]

I should've pointed out this is in serial

[GiudGiud]

I think it's gmres that is building a very large krylov base in this example. How many linear iterations is this case doing per non-linear iteration?
The LU sub-preconditioner also may not help.
Without profiling for memory it's hard to give you a definite answer. Thanks for reporting this.

[makeclean]

I tried hypre euclid, but that was a no go. See the image below, its the same problem once at a default resolution, and the 2nd refined further, results from gperf

[makeclean]

18 linear iterations per non-linear iteration, 3 non-linear iterations for convergence

[GiudGiud]

ok it's clear from gperf that the gradient caching on each face is the culprit.
Maybe we should re-consider that choice. I ll bring it up at the next FV meeting

Thanks!

[lindsayad]

I like that you said this was a quick one 😆 So yea @GiudGiud is working on this in in #18009 and in #18012. You did point out that the OpenFOAM to MOOSE comparison is apples to elephants and that's good. Because after eliminating the caching @GiudGiud 's profiles show that at least 60% of remaining memory allocation comes out of PETSc. 34% of that is a matrix alone and there's a pretty good chance that the referenced OpenFOAM case is using explicit time integration. Some of the remaining PETSc memory will be that minimum of 30 GMRES vectors (perhaps more if you increase the restart) unless you are using a different -ksp_type. You will still have a mass matrix with explicit time integration but for our spatial discretization of the time derivative term the matrix is purely diagonal and we actually have non-PETSc-based methods for solving this (no explicit matrix storage nor GMRES iterations required!).

Really appreciate this discussion as it will seriously cut down on our FV memory usage. It would be good to do some more serious profile comparisons to OpenFOAM moving forward where we use the same solution methods because we want to be in the same ballpark for sure. I would be shocked if we ever "beat" them, but yea in the same ballpark is where I want us to be.

[makeclean]

Glad to be of use :) When they get approved I have someone at our end to do a thorough OpenFOAM/MOOSE FV comparison.

[lindsayad]

I will prepare myself to feel embarrassed then haha. But this will be great to help us to continue to grow/improve

[GiudGiud]

@makeclean this PR got merged if you still want to look at openFOAM vs moose FV.

We now don't cache face gradients or values by default.
We still cache element gradients everywhere and face values on extrapolated boundary conditions (velocity inlet, pressure outlet)

[GiudGiud]

Closing this. Situation has largely improved with the caching PRs.