snappyHexMesh: motorBike tutorial error/hangs with Scotch with default (-DSCOTCH_PTHREAD[_MPI]) flags

Summary

snappyHexMesh doesn't work properly when OpenFOAM is linked against a Scotch library built with default options

Steps to reproduce

• Start with Scotch (latest from major version 6 or 7) built without removing the default -DSCOTCH_PTHREAD (major version 6) or -DSCOTCH_PTHREAD_MPI (major version 7) compilation flag (for reference, link to the official explanation of these flags)
• Compile OpenFOAM against that Scotch
• Run the $FOAM_TUTORIALS/incompressible/pisoFoam/LES/motorBike/motorBike case

Example case

What is the current bug behaviour?

snappyHexMesh fails or gets stuck and doesn't make any progress.

What is the expected correct behavior?

motorBike case successfully runs to completion.

Relevant logs and/or images

Possible failure modes:

...
Refined mesh in = 0.01 s
After refinement surface refinement iteration 0 : cells:1308  faces:4332  points:1758
Cells per refinement level:
    0	1276
    1	32
Skipping balancing since max unbalance 0.0674847 is less than allowable 0.1

Surface refinement iteration 1
------------------------------

Marked for refinement due to surface intersection          : 21 cells.
Marked for refinement due to curvature/regions             : 0 cells.
Determined cells to refine in = 0 s
Selected for refinement : 26 cells (out of 1308)
Edge intersection testing:
    Number of edges             : 4992
    Number of edges to retest   : 917
    Number of intersected edges : 110
Refined mesh in = 0.01 s
After refinement surface refinement iteration 1 : cells:1490  faces:4992  points:2069
Cells per refinement level:
    0	1267
    1	87
    2	136
[warn] event_base_loop: reentrant invocation.  Only one event_base_loop can run on each event_base at once.

Or (copied from here—thanks to @xuegy):

Cells per refinement level:
    0	1267
    1	87
    2	136
(4): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(5): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(6): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(7): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(0): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(1): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(2): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(3): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE

Environment information

• OpenFOAM version : v2212|v2112 (v2206 not tested, but I expect the same)
• Operating system : macOS
• Hardware info : arm64
• Compiler : clang

Possible fixes

• One could remove the -DSCOTCH_PTHREAD (major version 6) or -DSCOTCH_PTHREAD_MPI (major version 7) flags before compiling Scotch.

  • Unfortunately, this doesn't work for those using Scotch from a package manager. I note that Scotch maintainers recommend keeping those flags enabled, a recommendation that at least some packagers seem to be honoring (e.g. Debian, Homebrew). easy_build is also currently dealing with this issue with OpenFOAM.

• A workaround I found that seems to fix the problem is to change these lines of UPstream.C so as to force the call to MPI_Init_thread() to always use the MPI_THREAD_MULTIPLE option.

  • The error messages in the second (i.e. @xuegy's) log appear to point more directly at this being the cause.

EDIT: add link to easy_build issue

The UPstream is something we do not want to do. Initialising MPI with MPI_THREAD_MULTIPLE can lead to measurably slowdowns.

@Mattijs thanks. I only tried the UPstream workaround to see if that would fix the errors (to try to tell if there was actually a bug), so I had to list it as a possible fix. I understand that this change might be overkill, but I didn't trace the bug further than that.

If UPstream is not the source of the bug, then the bug must be in calling code that really needs MPI_THREAD_MULTIPLE (as seems to happen with the motorBike tutorial) incorrectly giving false as needsThread when initializing with UPstream::init().

The problem is scotch being built with thread support. snappyHexMesh does not need it, neither do any of the solvers. snappyHexMesh does not know that the scotch library selected requires a MPI_THREAD_MULTIPLE. I guess precompiled scotches should be in two variants - with and without MPI thread support.

The problem is scotch being built with thread support. snappyHexMesh does not need it, neither do any of the solvers.

Thanks a lot for clarifying this. I was wrongly assuming that snappyHexMesh setting needsThread to false—which eventually triggers this error if Scotch has MPI thread support—might have been (evidence of) a bug in OpenFOAM's logic.

snappyHexMesh does not know that the scotch library selected requires a MPI_THREAD_MULTIPLE.

What about having OpenFOAM test for this Scotch feature at build or runtime? At least with the point of catching this problem before it causes an error or hang (but at that point, it could force initializations to use MPI_THREAD_MULTIPLE if MPI thread support is enabled...)

The main problem is that homebrew being too aggressive to go with new version. And the maintainers are really double standard.

Many years ago, scotch was built without -DSCOTCH_PTHREAD like other Linux distro. It was reverted when someone submitted a recipe for newer version and no question asked. When I tried to submit a new recipe to remove the flag again, they made things really difficult. I spent like 2 weeks to persuade them and got my PR approved.

Recently they changed the recipe of openmp just because someone complained when using gcc on macOS and it caused huge mess when so many software failed on macOS.

If they continue doing this, eventually you’ll have to do create your own brew taps for all third party libraries, including openmpi (version 5.0 will be available soon).

As long as Scotch maintainers recommend that downstream devs/packagers leave -DSCOTCH_PTHREAD[_MPI] enabled, I think this problem may grow to potentially affect all sorts of packaged Scotch builds, as package managers update their own versions/recipes.

Besides the discussion at Debian I linked in the main body of the issue, here's also a link to an easy_build issue where they're too seeing this problem when compiling Scotch and OpenFOAM. My point is, I can see this issue affecting more than just Homebrew/macOS users in the near-ish future.

I still think there might be room for one more attempt at asking Homebrew core to disable this feature in their builds, but going against upstream recommendations is not the easiest.

b.t.w. do you have contact with the Scotch maintainers? Was trying to submit bugreport (unrelated to this) but cannot find a way of adding an issue.

Not really.

Looks like an account is needed to open an issue but I don't see a way to register there.

• workaround: use a fileHandler that forces mpi-threading (in mpi) (see e.g. OpenFOAM/global/fileOperations/collatedFileOperation/collatedFileOperation.H). Changes also the output format.
• workaround: add argList option to enforce mpi-threading and have snappyHexMesh,redistributePar etc. enforce it.
• as mentioned we saw a serious slowdown in big parallel runs - on the order of 20% if I remember correctly.

Q: the motorBike tutorial meshes with hierarchical decompositionMethod doesn't it? Meshing is general better with hierarchical - less mesh change in between refinement levels

A: Yes, decomposePar uses hierarchical.

added bug community labels

Could go weird and add an -mpi-thread option (much like the -parallel option), but the performance issue will still be there. The MPI + threaded as a bad idea is noted in various other places (not just OpenFOAM).

@Mattijs and @mark Thanks! If I get this right, there is no possible fix that can be made in OpenFOAM that doesn't noticeably hurt performance without requiring changes to the (recommended as of now) compilation flags of Scotch?

If so, maybe it's a problem that the Scotch folks should be aware of? As much as I'd like to see a workaround to avoid these unexpected runtime errors/hangs, a significant performance hit (especially if silent) also seems suboptimal.

I want to clarify that this not urgent for me as I'm custom building Scotch without MPI threading support for now, but I have been aware of this problem for a while on macOS/Darwin, and if more packagers follow upstream recommendations it should eventually hit the major Linux distros.

changed the description

We've warned the scotch maintainers about this. Can you see if the distro people are willing to e.g. build two versions?

@mark: affects only foamyMesh/snappyHexMesh/renumberMesh/decomposePar/redistributePar (until we add run-time balancing). Hardcode them to use mpi-thread?

We've warned the scotch maintainers about this

Do you have an issue link to share so that interested parties like myself can track it?

Can you see if the distro people are willing to e.g. build two versions?

Not sure if this is directed at me or someone else. Just in case, I'm not connected at all with Linux distro packagers. I could try to get Homebrew to change the build flags they use by asking again in the future (which might be viable assuming there's some consensus among other packagers and/or upstream changes their recommendations on default build flags).

mentioned in commit f8d6c4dab4fb73a1d3227ec44ccd919b9842a807

build issue reported under:

https://gitlab.inria.fr/scotch/scotch/-/issues/33

I've just tested with OpenFOAM 11 with scotch v7.0.4, and I confirm that this release fix the issue.

@mmercier @Mattijs Looks to be fixed with this distribution too (i.e. v2312 with Scotch 7.0.4).

Thanks for the update.

mentioned in commit 4412566c