snappyHexMesh: motorBike tutorial error/hangs with Scotch with default (-DSCOTCH_PTHREAD[_MPI]) flags

Summary

snappyHexMesh doesn't work properly when OpenFOAM is linked against a Scotch library built with default options

Steps to reproduce

• Start with Scotch (latest from major version 6 or 7) built without removing the default -DSCOTCH_PTHREAD (major version 6) or -DSCOTCH_PTHREAD_MPI (major version 7) compilation flag (for reference, link to the official explanation of these flags)
• Compile OpenFOAM against that Scotch
• Run the $FOAM_TUTORIALS/incompressible/pisoFoam/LES/motorBike/motorBike case

Example case

What is the current bug behaviour?

snappyHexMesh fails or gets stuck and doesn't make any progress.

What is the expected correct behavior?

motorBike case successfully runs to completion.

Relevant logs and/or images

Possible failure modes:

...
Refined mesh in = 0.01 s
After refinement surface refinement iteration 0 : cells:1308  faces:4332  points:1758
Cells per refinement level:
    0	1276
    1	32
Skipping balancing since max unbalance 0.0674847 is less than allowable 0.1

Surface refinement iteration 1
------------------------------

Marked for refinement due to surface intersection          : 21 cells.
Marked for refinement due to curvature/regions             : 0 cells.
Determined cells to refine in = 0 s
Selected for refinement : 26 cells (out of 1308)
Edge intersection testing:
    Number of edges             : 4992
    Number of edges to retest   : 917
    Number of intersected edges : 110
Refined mesh in = 0.01 s
After refinement surface refinement iteration 1 : cells:1490  faces:4992  points:2069
Cells per refinement level:
    0	1267
    1	87
    2	136
[warn] event_base_loop: reentrant invocation.  Only one event_base_loop can run on each event_base at once.

Or (copied from here—thanks to @xuegy):

Cells per refinement level:
    0	1267
    1	87
    2	136
(4): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(5): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(6): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(7): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(0): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(1): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(2): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE
(3): ERROR: SCOTCH_dgraphInit: Scotch compiled with SCOTCH_PTHREAD and program not launched with MPI_THREAD_MULTIPLE

Environment information

• OpenFOAM version : v2212|v2112 (v2206 not tested, but I expect the same)
• Operating system : macOS
• Hardware info : arm64
• Compiler : clang

Possible fixes

• One could remove the -DSCOTCH_PTHREAD (major version 6) or -DSCOTCH_PTHREAD_MPI (major version 7) flags before compiling Scotch.

  • Unfortunately, this doesn't work for those using Scotch from a package manager. I note that Scotch maintainers recommend keeping those flags enabled, a recommendation that at least some packagers seem to be honoring (e.g. Debian, Homebrew). easy_build is also currently dealing with this issue with OpenFOAM.

• A workaround I found that seems to fix the problem is to change these lines of UPstream.C so as to force the call to MPI_Init_thread() to always use the MPI_THREAD_MULTIPLE option.

  • The error messages in the second (i.e. @xuegy's) log appear to point more directly at this being the cause.

EDIT: add link to easy_build issue