Job is running very slowly

[dewan1988]

Dear MOOSE experts
I am running my job in a cluster for coarse grid CFD simulation with transient scenario and incompressible fluid. I am using 3D domain, and moose FV method. The number of elements in my current domain is 123 k and the mesh is unstructured. I have submitted my job with 2 nodes and 90 processors. But the job is converging very slowly (1 time-steps/hr).

How can I speed up the simulation? Could you please give me some suggestions?

[GiudGiud]

Hello

With MOOSE NSFV , you should be using a solve type of Newton not PJFNK.
With only 123k elements, so 123k * 4 degrees of freedom, 180 processes is too much. A rule of thumb is you start losing scalability below 20k degrees of freedom. Maybe try only 90 processes and only 48 processes and see if it runs faster this way

Once this is done, the next step is to work on the preconditioning rather than use the simple LU direct solve

Guillaume

[GiudGiud]

are you using RhieChow velocity interpolation and upwinding?

[dewan1988]

Dear GiudGiud
Thank you for your kind reply. Yes, I am using RhieChow velocity interpolation and upwinding. As per your suggestions, I have changed the preconditioning part:

Now, the linear iteration passing so fast but the non linear iteration part is slow and it's not converging

Currently, I am using 24 cores also.
Should I make any change in the executioner part?

[GiudGiud]

Hello

You can paste snippets between triple quotes instead of screenshots
```

pc_mat_solver_package goes in addition to the pc_type and the pc_factor_shift options

you can try to turn off the line search. There's advice on these convergence issues there
https://mooseframework.inl.gov/moose/application_usage/failed_solves.html

[dewan1988]

I have changed the preconditioning part in this way

[Preconditioning]                                                     
  [./SMP_NEWTON]                                                          
    type = SMP
    full = true
    solve_type = 'NEWTON'
    petsc_options_iname = '-pc_mat_solver_package -pc_type -pc_factor_shift_type'
    petsc_options_value = 'mumps lu NONZERO'
[../]

But still it is very slow

I am using this mesh file.

[GiudGiud]

LU does not scale to larger meshes. LU is a resilient and easy way to solve smaller sized problems. Once the problem gets big enough, you can see very large slowdowns and often run out of memory.
To get performance you ll want to examine the state of the art for solving these equations and set up the petsc options to use the desired preconditioners.

For incompressible flow, we are working on designing a field split preconditioning that does scale. There's an example in
modules/navier_stokes/tests/finite_volume/ins/channel-flow/2d-rc-no-slip.i

If it works for you great. If not, then you'll need to find the right combination of options for your problem.

Our other lead is to use segregated solvers instead. We have implemented a SIMPLE scheme in MOOSE, but it's limited to steady problems for now. We are about to incorporate an optimized linear system solver for it that will solve a lot of these issues. It's all rather on the edge, you can try it if you are comfortable with C++.

[dewan1988]

Dear GiudGiud
I am using the following PETSC option and values and executioner. The case initially converged 160 time steps in 12 hours and then when I restarted my simulation from the previous time steps it converges 108 time steps in 12 hours. Is it okay? Or are there any suggestions to increase the convergence rate? Am I making any mistake to select the PETSC option and values? Total elements in my domain is 123K and I am using 1 node and 64 processors to make the faster convergence, though 24 processors are enough for this case. I ran this simulation with 24, 48 processors also, I have found double time steps converges for 60 processors than 24, and for 48 it is 1.5 times higher than 24 processors. I have also tried to turn off the line search, but it's not working here.

[Preconditioning] 
  [./SMP_NEWTON]                          
    type = SMP
    full = true
    solve_type = 'NEWTON'
    petsc_options_iname = '-pc_mat_solver_package -pc_type -pc_factor_shift_type'
    petsc_options_value ='mumps ilu NONZERO'
[]
and following executioner:
[Executioner]                              
  type = Transient
  [./TimeStepper]
    type = IterationAdaptiveDT           
    growth_factor = 1.5
     optimal_iterations = 8                 
    linear_iteration_ratio = 50             
      dt =0.001                               
    cutback_factor = 0.8                     
    cutback_factor_at_failure = 0.8       
  [../]
 dtmin = 1e-6                                   
  dtmax = 0.03
 nl_rel_tol = 1e-4                            
 nl_abs_tol = 1e-4
 nl_max_its = 150
  l_tol = 1e-4                                   
 l_max_its = 100
 start_time = 7.79978                         
 end_time  = 10000                            
 #end_time  = 4000
 num_steps = 10000
[]

[dewan1988]

when I started the simulation for the case of 400k using the UOsolution of the case 120k or 50k it converged for some time steps (for the viscosity 1.05e-3) then it stopped converging. then I incresed the viscosity 1.05e-3 to 1.05 then ramp down to 0.105, 0.0105 and 0.00105, it didn't converged any more. No I didn't turn on the orthogonal correction. How can I add this orthogonal correction?

[GiudGiud]

oh not orthogonality correction. that is always active.
Skewness correction.
It's turned on in the finite volume advection kernels by specifying the advected quantity face interpolation method to skewness-corrected

[GiudGiud]

then ramp down to 0.105, 0.0105 and 0.00105

at what point in the ramp did it stop converging?
could you slow down the ramp to stay longer at that point?

[dewan1988]

It didn't converge in any case. But it converged for several time steps when I started the simulation with 0.00105 by using the solutionUO of the case 120K or 50K and increasing the tolerance (l &nl = e-6 &e-5 to e-4 &e-4 respectively).

[dewan1988]

It would be the cause that the element in the inlet pipe is more refined, because I found this scenario in one of my new mesh file where I generated the domain and the mesh by using the Gmsh tool. I found that my mesh file is working for the larger global size factor of the elements and but when I lowered the global mesh size factor to refine the case then it didn't work after certain global mesh size factor because the inlet mesh became so refined, which was not possible to simulate by moose. then I make the inlet section of the domain by changing some parameters then it worked.

[lindsayad]

I'm going to start a new thread because the previous one is massive and contains a conda tangent.

My first question is: what is your Reynolds number like? This will influence what type of precondtioning/solver method you can apply

[lindsayad]

What does a typical time step's solve look like? Please use backticks to create code blocks

[dewan1988]

@lindsayad, following is the solution of a single time step.
Time Step 54, time = 0.669691, dt = 0.061222
Pre-SMO residual: 0.00793607
0 Nonlinear |R| = 7.936072e-03
0 Linear |R| = 7.936072e-03
1 Linear |R| = 2.755572e-03
2 Linear |R| = 2.489114e-04
3 Linear |R| = 8.918983e-05
4 Linear |R| = 3.903350e-05
5 Linear |R| = 2.321761e-05
6 Linear |R| = 1.570184e-05
7 Linear |R| = 1.177219e-05
8 Linear |R| = 8.527621e-06
9 Linear |R| = 6.361247e-06
10 Linear |R| = 5.009492e-06
11 Linear |R| = 4.198449e-06
12 Linear |R| = 3.617002e-06
13 Linear |R| = 3.226332e-06
14 Linear |R| = 2.971982e-06
15 Linear |R| = 2.877857e-06
16 Linear |R| = 2.730112e-06
17 Linear |R| = 2.371489e-06
18 Linear |R| = 2.099327e-06
19 Linear |R| = 1.779490e-06
20 Linear |R| = 1.549898e-06
21 Linear |R| = 1.282474e-06
22 Linear |R| = 7.987862e-07
23 Linear |R| = 5.283119e-07
24 Linear |R| = 3.874162e-07
25 Linear |R| = 2.751974e-07
26 Linear |R| = 2.354450e-07
27 Linear |R| = 2.224174e-07
28 Linear |R| = 2.161625e-07
29 Linear |R| = 2.149922e-07
30 Linear |R| = 2.149620e-07
31 Linear |R| = 2.149618e-07
32 Linear |R| = 2.149583e-07
33 Linear |R| = 2.149477e-07
34 Linear |R| = 2.149434e-07
35 Linear |R| = 2.149425e-07
36 Linear |R| = 2.149329e-07
37 Linear |R| = 2.148642e-07
38 Linear |R| = 2.147871e-07
39 Linear |R| = 2.146930e-07
40 Linear |R| = 2.146689e-07
41 Linear |R| = 2.146659e-07
42 Linear |R| = 2.146480e-07
43 Linear |R| = 2.146332e-07
44 Linear |R| = 2.145068e-07
45 Linear |R| = 2.143169e-07
46 Linear |R| = 2.140054e-07
47 Linear |R| = 2.137964e-07
48 Linear |R| = 2.134966e-07
49 Linear |R| = 2.131924e-07
50 Linear |R| = 2.121424e-07
51 Linear |R| = 2.078384e-07
52 Linear |R| = 1.977328e-07
53 Linear |R| = 1.643942e-07
54 Linear |R| = 1.167995e-07
55 Linear |R| = 7.513696e-08
1 Nonlinear |R| = 5.371659e-03
0 Linear |R| = 5.371659e-03
1 Linear |R| = 2.152574e-03
2 Linear |R| = 2.117357e-04
3 Linear |R| = 7.037866e-05
4 Linear |R| = 3.111360e-05
5 Linear |R| = 1.800756e-05
6 Linear |R| = 1.209162e-05
7 Linear |R| = 8.956012e-06
8 Linear |R| = 6.348354e-06
9 Linear |R| = 4.671962e-06
10 Linear |R| = 3.632113e-06
11 Linear |R| = 2.999884e-06
12 Linear |R| = 2.542629e-06
13 Linear |R| = 2.239342e-06
14 Linear |R| = 2.058640e-06
15 Linear |R| = 1.995802e-06
16 Linear |R| = 1.918659e-06
17 Linear |R| = 1.672280e-06
18 Linear |R| = 1.474329e-06
19 Linear |R| = 1.248174e-06
20 Linear |R| = 1.096758e-06
21 Linear |R| = 9.154800e-07
22 Linear |R| = 5.793961e-07
23 Linear |R| = 3.893507e-07
24 Linear |R| = 2.839240e-07
25 Linear |R| = 1.986226e-07
26 Linear |R| = 1.671127e-07
27 Linear |R| = 1.561103e-07
28 Linear |R| = 1.509873e-07
29 Linear |R| = 1.500069e-07
30 Linear |R| = 1.499917e-07
31 Linear |R| = 1.499917e-07
32 Linear |R| = 1.499890e-07
33 Linear |R| = 1.499799e-07
34 Linear |R| = 1.499753e-07
35 Linear |R| = 1.499748e-07
36 Linear |R| = 1.499658e-07
37 Linear |R| = 1.499078e-07
38 Linear |R| = 1.498348e-07
39 Linear |R| = 1.497643e-07
40 Linear |R| = 1.497552e-07
41 Linear |R| = 1.497548e-07
42 Linear |R| = 1.497517e-07
43 Linear |R| = 1.497471e-07
44 Linear |R| = 1.496584e-07
45 Linear |R| = 1.495224e-07
46 Linear |R| = 1.492976e-07
47 Linear |R| = 1.491667e-07
48 Linear |R| = 1.489239e-07
49 Linear |R| = 1.487431e-07
50 Linear |R| = 1.478794e-07
51 Linear |R| = 1.455867e-07
52 Linear |R| = 1.397970e-07
53 Linear |R| = 1.193011e-07
54 Linear |R| = 8.658228e-08
55 Linear |R| = 5.442514e-08
56 Linear |R| = 3.577573e-08
2 Nonlinear |R| = 2.310320e-03
0 Linear |R| = 2.310320e-03
1 Linear |R| = 1.037535e-04
2 Linear |R| = 1.412109e-05
3 Linear |R| = 4.377150e-06
4 Linear |R| = 1.988819e-06
5 Linear |R| = 1.094745e-06
6 Linear |R| = 7.003765e-07
7 Linear |R| = 4.550137e-07
8 Linear |R| = 2.925521e-07
9 Linear |R| = 1.971289e-07
10 Linear |R| = 1.318675e-07
11 Linear |R| = 9.674303e-08
12 Linear |R| = 7.260223e-08
13 Linear |R| = 5.628509e-08
14 Linear |R| = 4.661584e-08
15 Linear |R| = 4.356202e-08
16 Linear |R| = 4.137240e-08
17 Linear |R| = 3.791217e-08
18 Linear |R| = 3.519399e-08
19 Linear |R| = 3.151400e-08
20 Linear |R| = 2.898091e-08
21 Linear |R| = 2.521020e-08
22 Linear |R| = 2.072345e-08
3 Nonlinear |R| = 1.648891e-05
0 Linear |R| = 1.648891e-05
1 Linear |R| = 1.378066e-06
2 Linear |R| = 1.445614e-07
3 Linear |R| = 3.646037e-08
4 Linear |R| = 1.951068e-08
5 Linear |R| = 1.672635e-08
6 Linear |R| = 1.565961e-08
7 Linear |R| = 1.522822e-08
8 Linear |R| = 1.476997e-08
9 Linear |R| = 1.441115e-08
10 Linear |R| = 1.384423e-08
11 Linear |R| = 1.297924e-08
12 Linear |R| = 1.173304e-08
13 Linear |R| = 1.025172e-08
14 Linear |R| = 8.908641e-09
15 Linear |R| = 7.846652e-09
16 Linear |R| = 7.511634e-09
17 Linear |R| = 7.329518e-09
18 Linear |R| = 7.257570e-09
19 Linear |R| = 7.234636e-09
20 Linear |R| = 7.227560e-09
21 Linear |R| = 7.217352e-09
22 Linear |R| = 7.184781e-09
23 Linear |R| = 7.167217e-09
24 Linear |R| = 7.163254e-09
25 Linear |R| = 7.153134e-09
26 Linear |R| = 7.108224e-09
27 Linear |R| = 7.020037e-09
28 Linear |R| = 6.775274e-09
29 Linear |R| = 6.133723e-09
30 Linear |R| = 4.670658e-09
31 Linear |R| = 3.945391e-09
32 Linear |R| = 3.471780e-09
33 Linear |R| = 3.273104e-09
34 Linear |R| = 3.089447e-09
35 Linear |R| = 2.977621e-09
36 Linear |R| = 2.934418e-09
37 Linear |R| = 2.864268e-09
38 Linear |R| = 2.834157e-09
39 Linear |R| = 2.811701e-09
40 Linear |R| = 2.808687e-09
41 Linear |R| = 2.808193e-09
42 Linear |R| = 2.803674e-09
43 Linear |R| = 2.798858e-09
44 Linear |R| = 2.786982e-09
45 Linear |R| = 2.782442e-09
46 Linear |R| = 2.781110e-09
47 Linear |R| = 2.780495e-09
48 Linear |R| = 2.780412e-09
49 Linear |R| = 2.779556e-09
50 Linear |R| = 2.777879e-09
51 Linear |R| = 2.772376e-09
52 Linear |R| = 2.765887e-09
53 Linear |R| = 2.750714e-09
54 Linear |R| = 2.697276e-09
55 Linear |R| = 2.541699e-09
56 Linear |R| = 2.161808e-09
57 Linear |R| = 1.862531e-09
58 Linear |R| = 1.425345e-09
59 Linear |R| = 1.134934e-09
60 Linear |R| = 8.116414e-10
61 Linear |R| = 6.734633e-10
62 Linear |R| = 5.704222e-10
63 Linear |R| = 4.380400e-10
64 Linear |R| = 3.246971e-10
65 Linear |R| = 2.696393e-10
66 Linear |R| = 2.429193e-10
67 Linear |R| = 2.238164e-10
68 Linear |R| = 2.118089e-10
69 Linear |R| = 2.052268e-10
70 Linear |R| = 2.013866e-10
71 Linear |R| = 2.006392e-10
72 Linear |R| = 1.998965e-10
73 Linear |R| = 1.994812e-10
74 Linear |R| = 1.991883e-10
75 Linear |R| = 1.990412e-10
76 Linear |R| = 1.990121e-10
77 Linear |R| = 1.985506e-10
78 Linear |R| = 1.984032e-10
79 Linear |R| = 1.983576e-10
80 Linear |R| = 1.982347e-10
81 Linear |R| = 1.982326e-10
82 Linear |R| = 1.982201e-10
83 Linear |R| = 1.981510e-10
84 Linear |R| = 1.981099e-10
85 Linear |R| = 1.980806e-10
86 Linear |R| = 1.975740e-10
87 Linear |R| = 1.958138e-10
88 Linear |R| = 1.925315e-10
89 Linear |R| = 1.811159e-10
90 Linear |R| = 1.553111e-10
4 Nonlinear |R| = 1.162941e-08
Solve Converged!
Finished Solving

[lindsayad]

that solve really doesn't look too bad. It just sounds like it's very slow? How long for a single timestep?

[dewan1988]

@lindsayad
Earlier it was very slow because I did use different preconditioning options. Then GiudGiud recommended some preconditioning options what I am using now. Then it becomes faster then previous. In my previous simulation I used 64 cores only for 50k elements without proper calculations. Now I use the number of processors/cores by calculating as follows: no_elements*4 dfs/20,000. Earlier I got only 175 time_steps solve in 12 hrs and 64 cores but now I am getting more than 175 time steps solves with only 6 cores/processors.

[lindsayad]

What I will say is that solving transient 3D highish Reynolds number problems with a decent number of elements is an active area of work for us. As @GiudGiud mentioned earlier @grmnptr has developed steady segregated solvers that are doing quite well for moderately fine meshes with finite volume. I have shown some nice results using finite elements for high Reynolds and fine meshes. But we do not yet have a great solver for transient, high Reynolds, medium fidelity mesh, finite volume