Update PETSc to 3.20.3, update libmesh

[cticenhour]

Closes #26461

[cticenhour]

@roystgnr I'd like to get one round of testing in on this before piling more on, but would you be interested in a libMesh update going in alongside this?

[moosebuild]

Job Test timings on fbd5908 wanted to post the following:

View timings here

This comment will be updated on new commits.

[moosebuild]

Job Documentation on fbd5908 wanted to post the following:

View the site here

This comment will be updated on new commits.

[roystgnr]

would you be interested in a libMesh update going in alongside this?

Probably not. I've got a minor fix that would be useful in MOOSE, but not worth the hassle alone, and I've got a major branch that isn't likely to be ready to merge before the end of the week.

[cticenhour]

Alright - will ping again if this ends up lingering through Friday or Monday.

[moosebuild]

Job Coverage on fbd5908 wanted to post the following:

Framework coverage

	c581c9	#26462 fbd590
	Total	Total	+/-	New
Rate	85.10%	85.10%	-0.00%	-
Hits	98663	98662	-1	0
Misses	17273	17274	+1	0

Diff coverage report

Full coverage report

Modules coverage

Coverage did not change

Full coverage reports

Reports

• framework
• chemical_reactions
• combined
• contact
• electromagnetics
• external_petsc_solver
• fluid_properties
• fsi
• functional_expansion_tools
• geochemistry
• heat_transfer
• level_set
• misc
• navier_stokes
• optimization
• peridynamics
• phase_field
• porous_flow
• ray_tracing
• rdg
• reactor
• richards
• scalar_transport
• solid_properties
• stochastic_tools
• tensor_mechanics
• thermal_hydraulics
• xfem

This comment will be updated on new commits.

[lindsayad]

I would be interested in a libmesh update

[lindsayad]

Can't build #26477 without a libmesh update. A couple cool new things in libMesh directly leveraged in that MOOSE PR are L2_HIERARCHIC_VEC and the ability to form a callback post linear-solve, right before the line search.

That being said, I would be willing to wait for @roystgnr's major branch to merge (is it libMesh/libmesh#3759 or something else?)

[roystgnr]

Those are pretty good reasons for the update.

I don't want to rush libMesh/libmesh#3759, though that is what I was thinking about. The current status of libMesh::MeshBase is roughly "what's a class invariant", and I'm trying to figure out if I can efficiently change that to "you can know a priori that the mesh is prepared except for a runtime-testable and easily-individually-correctable set of exceptions", but if I stop here then we're just at "what's a class invariant, but more confusing".

So we should probably update the submodule after all, and do it without that. I'd like libMesh/libmesh#3758 to make it to master, and I do have one other PR that I'm hoping to get in today too, but there's actually more than enough to justify an update as-is, now that I look over the git logs. If we wait much longer getting libMesh/libmesh#3736 downstream I'll probably forget entirely about the MOOSE side of it...

[roystgnr]

I've pushed newsletter docs to roystgnr@1bfae7d if you want to cherry-pick that to go alongside a libmesh submodule update.

[cticenhour]

Thanks for your input! I will update this PR tomorrow to include a libMesh update as well. The only thing holding this back on testing at the moment is a FBLASLAPACK download issue during PETSc alt container creation....working with Logan on it.

[roystgnr]

If there's more futzing to be done anyway, it wouldn't hurt to get libMesh up to libMesh/libmesh@3fcc7e7 - the additional changelog entries would be:

• Fixes for VTK .pvtu reading, and preservation of libMesh node, element, and subdomain ids written to VTK files
• Elem::volume() no longer can throw an exception when called on an element without an invertible mapping; hence print_info() and get_info() are now safe to use when emitting debugging messages about twisted geometric elements.

IMHO neither of those are super high priority if you just want to focus on getting the PETSc issues fixed and getting this in.

[cticenhour]

@roystgnr The GCC min / PETSc-alt failures are all tied to the FBLASLAPACK download issue (which is preventing the build of a new container for those tests), but the new app failures I am seeing are due to the libMesh additions. Do you mind taking a look today? They all seem to be related to the nodeset changes or similar.

[roystgnr]

Will do! Looks like we've got a segfault in a Malamute test and 7 (looks like 2 independent) exodiffs in Bison tests?

[roystgnr]

The good news is that I understand the MALAMUTE failure now.

The bad news is that the MALAMUTE failure is coming from MOOSE doing something unsupported and, up until now, getting away with it via dumb luck.

That input deck creates a mesh of QUAD8 elements. It then has MOOSE create Edge3 elements along one side of some of them, because formulation = mortar, and MOOSE manually does a edge3->set_interior_parent(quad8) on each. It then has MOOSE convert all the Quad8 elements to Quad9, because second_order = true, and the trouble with this of course is that you can't actually convert one class into another in C++. What libMesh does to "convert" a mesh like this is to delete all the lower-order elements and allocate higher-order elements to replace them. This has apparently (probably?) been working out for MALAMUTE in the past because the allocated elements have been ending up at the same addresses in memory!? And then those interior_parent pointers stay valid. But nothing guarantees that happening, and any kind of modification of the all_second_order code (much less my recent extensive modifications) can stop it from happening. (as could a change of compiler, or of compiler flags, or of day of the week...)

I'm trying to figure out a way to make this robust.

[lindsayad]

Where is MOOSE doing this? Doesn’t moose rely on libMesh’s api for converting the mesh to second order?

[lindsayad]

Haven’t you been doing a bunch of work in these mesh order upgrade methods? Is it possible the behavior is changing in this update? It would seem like quite a coincidence for this to have been working for all CI runs for months

[roystgnr]

Where is MOOSE doing this? Doesn’t moose rely on libMesh’s api for converting the mesh to second order?

The set_interior_parent() calls are in LowerDBlockFromSidesetGenerator, I believe. The all_second_order() call is in SetupMeshAction. It is using the libMesh API, but the problem is that even the libMesh API can't guarantee that deallocation and reallocation of objects gives you back the same pointers if only you do it in just the right order. Maybe there's a way to do it with fancy C++ tricks, if we destruct the existing Elem without freeing its memory and then you use placement-new to put the new Elem in its place (assuming all Elem subclasses are exactly the same size, which maybe?), but we don't do that. We just delete the existing Elem and new a replacement, and the memory allocator has apparently been returning the same pointer, but it's not at all guaranteed to.

Haven’t you been doing a bunch of work in these mesh order upgrade methods? Is it possible the behavior is changing in this update? It would seem like quite a coincidence for this to have been working for all CI runs for months

It's definitely triggered by this update (something is performing another allocation in between the delete and the new? I'm now doing the new before the delete?), but it's not caused by this update. I like the "nasal demons" joke about Undefined Behavior, where a compiler is allowed to respond to UB by making demons come out your nose, but really the most insidious allowed response to UB is the one we're seeing here: it's allowed to work the way you hoped right up until for no apparent reason it stops working.

On the bright side, the refactoring should at least make it possible to fix every case of this all in one place.

On the not-so-bright side, the fix is going to make some of my new capabilities asymptotically slower. We can't safely upgrade O(N_range) elements until after we first check O(N_elem) elements to see if they have any interior_parent() links that need to be moved as part of the upgrade. I don't think anybody's going to need them to be O(N_range) but it would have been nice to have had the option.

[roystgnr]

I can't speak to the PETSc part of the change, but everything on the libMesh side should be ready to go.

[loganharbour]

Going to add in a minor apptainer change and one more libmesh bump.

[roystgnr]

everything on the libMesh side should be ready to go.

I'd like to thank Nemesis for only striking me down with a new bug report from Alex, and not with the lightning bolt that my hubris above may have deserved.

I'm looking into it now.

[moosebuild]

Job Precheck on fbd5908 wanted to post the following:

The following file(s) are changed:

conda/libmesh/meta.yaml

The libmesh conda configuration has changed; ping @idaholab/moose-dev-ops

[cticenhour]

@milljm Could you please give us another set of eyes for these changes? Thanks!

[milljm]

For the Conda stuff, all looks well!

[roystgnr]

The fix for that last remaining libMesh bug is in libMesh/libmesh#3772

After we get that to master (hopefully tomorrow), we can update the submodule here and we'll have taken care of everything outstanding I know about.

I'm thinking my approval for merging as-is still stands, though, if anyone is chomping at the bit. The updates in this PR right now include fixes for potentially more serious and definitely more wide-ranging bugs, in long-standing use cases where as far as I can tell we've so far been saved from data corruption by kindly compilers and dumb luck, whereas the bug fixed in libMesh/libmesh#3772 only affects Tet14 refinement, a relatively new feature.

[lindsayad]

so we didn't get Roy's last fix in?

[roystgnr]

Nope; sorry. Hope you're okay working from a manual submodule update until next month.

[roystgnr]

Looks like we missed the LIBMESH_REV update in docker_ci/Dockerfile here? And yet that didn't cause any problems for users?

[loganharbour]

Looks like we missed the LIBMESH_REV update in docker_ci/Dockerfile here? And yet that didn't cause any problems for users?

I need to remove the docker_ci folder; it's not used anymore.