Dummy library such as Pstream is used on macOS

Summary

While executing in parallel mode, the dummy Pstream library is used. But there is correct Pstream library in

$FOAM_LIBBIN/$

FOAM_MPI.

Steps to reproduce

After building the ThirdPaty code, it is necessary for the following command.

for i in $FOAM_EXT_LIBBIN/*.dylib ;do install_name_tool -id $i $i done

After that, build the OpenFOAM code.

On the directory of tutorials/incompressible/icoFoam/cavityMappingTest.

./Allrun-parallel

Example case

In the "log.icoFOAM,

` --> FOAM FATAL ERROR: The dummy Pstream library cannot be used in parallel mode

    From function static bool Foam::UPstream::init(int &, char **&, const bool)
    in file UPstream.C at line 50.

FOAM exiting`

What is the current bug behaviour?

FOAM FATAL ERROR.

What is the expected correct behavior?

Normal exiting in parallel mode.

Relevant logs and/or images

Environment information

• OpenFOAM version : v1912
• Operating system : macOS 10.15.4
• Hardware info : MacBook Pro
• Compiler : Apple clang version 11.0.3 (clang-1103.0.32.29)

Possible fixes

Current macOS ignores the path of LD_LIBRARY_PATH and DYLD_LIBRARY_PATH. So dummy library was loaded instead of correct library.

On macOS, it has to use the library path explicitly. I attached the patch file.

parallel.patch

@mark

Hi @alexey - it this patch reasonable, or is there something else going on here?

I have taken a brief look at the patch and the way to reproduce the error and I think the problem is in System Integrity Protection (https://support.apple.com/en-us/HT204899). It clears DYDL_LIBRARY_PATH for bash scripts, so MPI version of Pstream is lost (which, in general, is used instead of dummy version because MPI library path goes before dummy in DYDL_LIBRARY_PATH). I solve this problem with:

# Work-around for cleared PATH and DYLD_LIBRARY_PATH variables
isSIPEnabled()
{
    $(which -s csrutil) && {
    [ $(csrutil status | awk '{print $5}' | tr -d ".") = "enabled" ] \
        && return 0
    }
    return 1
}

[ "$(uname -s)" = "Darwin" ] && isSIPEnabled && . $WM_PROJECT_DIR/etc/bashrc

in bin/tools/RunFunctions. It works, so I do not look for more elegant/correct solutions.

I have tried to reproduce the error and was not able to. My build works fine in serial and parallel modes.

@fski Could you, please, run the tutorial from command line (copy-pasting commands from Allrun-parallel file)? Though, I do not know, if you still have OpenFOAM(R) built without attached patch.

Hi @alexey I know about SPI(System Integrity Protection). While it is enabled, we can not set the environment value of DYLD_LIBRAEY_PATH and LD_LIBRARY_PATH. I can reproduce again, only modify the file of "src/OpenFOAM/Make/options. Generally SPI is enabled, so I enter the recovery mode, and change the SPI mode to disable. After that, I treid to reproduce. Of cource it is no problem. But I think that this mode is not normal because the security level is lower, maybe Apple does not recommend. I hope that Mac users of OpenFOAM increase, so I prefer that SPI is enabled.

By the way, I know that there is another problem while we can not use the environment value of DYLD_LIBRAEY_PATH. This problem is loading dynamic library such as fieldFunctionObjects. So I created the patch that it has a function in search path before call dlopen(). Please check this.

search-dlopen.patch

@fski I do not propose to disable SIP. I propose to re-setup environment after you start script. Usually Allrun scripts source RunFunctions file, so I have added environment re-setup to this file. You do not need LD_LIBRARY_PATH on macOS (see man dyld for a list of environment variables).

I would propose not to mix different issues in this one. I was not able to get neither the problem you have described in your second paragraph, nor the problem you are trying to solve with attached patch. Could you open separate issue and describe the problem in more details (with a way to reproduce it)? Also patch is not quite consistent, in one place you call variable FOAM_DYLD_LIBRARY_PATH, in another FOMA_DYLD_LIBRARY_PATH. Sure it is just a typo, still it is nice to avoid them in patches.

@alexey OK, I will open separate issue. Also, it is a typo. thank you for pointing that out.

This issue is caused by wrong DYLD_LIBRARY_PATH and has the same solution as #1664 (closed).

Hi @fski - would you mind to update the status of this ticket, please? Please feel free to close the ticket if the issue has been resolved for you.

mentioned in issue #1664 (closed)

mentioned in issue #2555 (closed)

Assuming resolved - please re-open if needed (see #2555 (closed))

closed