rpath compilation rules for Darwin

@mark @alexey Thanks for the work. If you give me a few days, I'm happy to review the code on this one.

To be honest upfront, I'm still not onboard with hardcoding the path to $FOAM_USER_LIBBIN in OpenFOAM binaries ($FOAM_SITE_LIBBIN I think is okay); but I haven't yet looked at the code changes and I promise to give it more thought.

I am not sure if it is worth messing with manual compilation of third party libraries. Some of them need special patches to compile with clang, some of them can only be compiled with gcc. And all third party build scripts (automake, cmake, custom scripts) has to be adapted to this approach. Theoretically we can edit install names of the compiled libraries with install_name_tool, yet, won't it be easier to symlink all third-party libraries into $FOAM_LIBBIN/third-party and add this folder to -rpath flag?

Actually, I meant just setting the paths to third-party libraries the same way in OpenFOAM binaries; I agree on not touching the third-party packages themselves at all (i.e., it's their problem if they are not relocatable). Your symlink suggestion I think is closer to what I'd actually want. Still, I think it's better to leave this suggestion of mine out of scope for this one. I'd definitely like to see it tackled too but it can wait until the basics are working.

might want to consider ${FOAM_EXT_LIBBIN} and ${FOAM_EXT_LIBBIN}/${FOAM_MPI} as well. These are the paths to the ThirdParty libraries (if any).
there is also ${FOAM_LIBBIN}/${FOAM_MPI} to consider - as @executable_path/../lib/${FOAM_MPI} ?

@mark Thank you for the suggestion, I will add *_EXT_* paths to linker flags (if these variables are available).

Concerning seconds point: ${FOAM_LIBBIN}/${FOAM_MPI} is @executable_path/../lib/${FOAM_MPI} for executables and @loader_path/${FOAM_MPI} for libraries.

Patch with added $FOAM_EXT_LIBBIN paths:

02-feature-darwin-rpath-wmake-rules.patch

mentioned in commit bcbae122e22882009c9635bff451bee76870705a

Hi @alexey (and @gerlero) - I've patched in the changes to the make rules (slightly reworked), but will initially delay the changes to foamJob and RunFunctions until we have 100% certainty.

Pushed to the issue-2948-MacOS-rpath branch for you both to test.

All right, I've finally had some time to try out a full build (I'm going by @mark's branch). Thanks again to @alexey for the implementation. My comments are:

First of all, unfortunately for some reason on my builds I see that only the dummy MPI path gets added to RPATH, and not the $FOAM_MPI subdir (despite the latter variable being defined), which is not right and paradoxically causes the error it is expected to avoid. I don't know what's causing this. @alexey Have you seen the same behavior in your builds? I'll make sure to post the output of otool below.
As for adding $FOAM_USER_LIBBIN as an rpath, I'm sorry, but I still think it's not the best design to hardcode any these paths (which depend on the user's name and $HOME) into the OpenFOAM binaries. It's unexpected for whoever is building/using it, and very different from how OpenFOAM works on other platforms. It also won't work (and may even break things?) whenever OpenFOAM is built by a different user, which includes my openfoam-app project. I have to ask: in which scenarios do first-party OpenFOAM binaries actually need to load libraries from $FOAM_USER_LIBBIN, and, is there a different way to accomplish this more reliably that doesn't hardcode the user paths in the binaries?
- I'd lean towards skipping $FOAM_USER_LIBBIN here altogether and relying on the existing $FOAM_LD_LIBRARY_PATH workaround (and/or using a non-Apple shell, both of which already do a good job) for the odd(?) case where an OpenFOAM binary needs to load a user's lib.
As for user projects (i.e., third-party projects installed to $FOAM_USER_*), I see that their compilation is covered by the same rules, so their paths get rewritten too (I've tried one of mine to confirm). I don't think this is strictly necessary, but I guess it makes sense to apply the same rules everywhere, and it doesn't seem to break anything here.
Regarding delaying the changes to foamJob and RunFunctions (which @mark has done already in this branch), I agree with @mark's and strongly recommend keeping that workaround with no expiration date for now, even after any of these changes are merged.

There is a typo in rules on line 24 in rpath file, which causes missing $FOAM_MPI. This
```
ifneq (,$(strip,$(FOAM_MPI)))
```
should be
```
ifneq (,$(strip $(FOAM_MPI)))
```
Could we introduce compilation flag in WM_COMPILE_CONTROL? Something like darwinrpathuserlibbin, upon setting which FOAM_USER_LIBBIN is added to/excluded from rpath. Alternatively prebuilt apps can use install_name_tool to correct rpath during first run.
Influence of rpath on user and site-installed libraries, in fact, is interesting and I will test this use-case more intensely. @loader_path in this case becomes $FOAM_USER_LIBBIN (or $FOAM_SITE_LIBBIN), and this can influence dynamic library search.
Wasn't the whole point of rpath getting rid of these work-arounds? Though we can still keep them, just in case.

fixing the typo (thanks!)

selection/deselection of rpath:

# Linking:
# with or without rpath components on MacOS
# - current default is with rpath unless explicitly disabled

ifneq (,$(findstring ~rpath,$(WM_COMPILE_CONTROL)))
    include $(DEFAULT_RULES)/link-c++
else
    include $(DEFAULT_RULES)/link-rpath-c++
endif

what is the reasonable preferred default for user libbin (and site libbin) though?

Influence of rpath on user and site-installed libraries, in fact, is interesting and I will test this use-case more intensely. @loader_path in this case becomes $FOAM_USER_LIBBIN (or $FOAM_SITE_LIBBIN), and this can influence dynamic library search.

Indeed, user-installed binaries are getting an rpath entry pointing to @loader_path (for libs) or @executable_path/../lib (for executables), which is effectively $FOAM_USER_LIBBIN. I'd expect a similar thing for site installs, but I haven't tested that (that should be tested before a merge). This is neat, but also effectively makes the addition of an explicit rpath entry pointing at $FOAM_USER_LIBBIN (or $FOAM_SITE_LIBBIN) redundant.

EDIT: format

@alexey As for your other points, I'm still unable to think of a scenario where a first-party OpenFOAM binary needs to load a user (or site) library. If you or @mark can give me an example of when/why this is necessary, then this might influence my opinion on if/how to deal with $FOAM_USER_LIBBIN as a possible rpath, and also whether the existing workaround $FOAM_LD_LIBRARY_PATH is still needed.

As far as I can tell, first-party binaries never get any "hardcoded" linker entries pointing at a $FOAM_USER_LIBBIN or $FOAM_SITE_LIBBIN when compiling OpenFOAM; and neither $FOAM_USER_LIBBIN nor $FOAM_SITE_LIBBIN are in $DYLD_LIBRARY_PATH/$FOAM_LD_LIBRARY_PATH. With that in mind, I don't see how a first-party binary attempting to load a user or site library would ever work on a default install (but I might be missing something...).

In the case that (preferably both of) you can confirm my intuition from above, then $FOAM_USER_LIBBIN and $FOAM_SITE_LIBBIN must not be set as rpaths. Also in that scenario this change can supersede the $FOAM_LD_LIBRARY_PATH workaround for all purposes (not saying the workaround should be dropped immediately, but it could be done at some point).

EDIT: $FOAM_DYLD_LIBRARY_PATH -> $FOAM_LD_LIBRARY_PATH

@gerlero - for general information. If you wish to control the use of user/site locations during compilation of applications (this is actually a very good idea) consider adding these lines to your build script:

# Avoid external influence on the environment
export FOAM_CONFIG_MODE="o"
unset FOAM_SETTINGS

This is part of our RPM builds, but oddly enough I don't seem to have it in the debian builds (not sure why it is missing there).

The FOAM_CONFIG_MODE can contain any of ugo - see comments in the header of bin/foamEtcFile

@mark not sure I follow. Does changing this option cause a first-party OpenFOAM binary (like icoFoam) to be able to load some user's library at $FOAM_USER_LIBBIN? Is this the default behavior, and is this a good idea?

As far as I can tell, the only point of setting $FOAM_USER_LIBBIN as an rpath is to support doing something like that (i.e., first-party or site-installed binary that wants to link against a user-installed library).

$FOAM_USER_LIBBIN and $FOAM_SITE_LIBBIN are in $FOAM_LD_LIBRARY_PATH (line 232 of etc/config.sh/setup):

# OpenFOAM user, group libraries
_foamAddLib  "$FOAM_USER_LIBBIN:$FOAM_SITE_LIBBIN"

What goes into FOAM_USER_LIBBIN and FOAM_SITE_LIBBIN? I can imagine the following use-cases:

User and site-supplied addition. Function object, fvOption, etc, which is loaded in case dictionaries. This use-case is handled by dlOpen, which goes through FOAM_LD_LIBRARY_PATH (if DYLD_LIBRARY_PATH is empty).
User and site-supplied overrides. We can imagine that user would like to use his own libturbulenceModels.dylib, case of site-provided libturbulenceModels.dylib looks even more realistic. Right now these overrides are done by prepending DYLD_LIBRARY_PATH with user and site folders.

The last use-case needs the folders in rpath.

$FOAM_USER_LIBBIN and $FOAM_SITE_LIBBIN are in $FOAM_LD_LIBRARY_PATH (line 232 of etc/config.sh/setup)

@alexey Interesting. They are not in $FOAM_LD_LIBRARY_PATH for me:

echo $FOAM_LD_LIBRARY_PATH

/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib/openmpi:/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib:/Volumes/OpenFOAM-rpath/usr/opt/fftw/lib:/Volumes/OpenFOAM-rpath/usr/opt/mpfr/lib:/Volumes/OpenFOAM-rpath/usr/opt/gmp/lib:/Volumes/OpenFOAM-rpath/usr/opt/cgal/lib:/Volumes/OpenFOAM-rpath/usr/opt/boost/lib:/Volumes/OpenFOAM-rpath/usr/opt/open-mpi/lib:/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib/dummy

nor in $DYLD_LIBRARY_PATH:

echo $DYLD_LIBRARY_PATH

/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib/openmpi:/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib:/Volumes/OpenFOAM-rpath/usr/opt/fftw/lib:/Volumes/OpenFOAM-rpath/usr/opt/mpfr/lib:/Volumes/OpenFOAM-rpath/usr/opt/gmp/lib:/Volumes/OpenFOAM-rpath/usr/opt/cgal/lib:/Volumes/OpenFOAM-rpath/usr/opt/boost/lib:/Volumes/OpenFOAM-rpath/usr/opt/open-mpi/lib:/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib/dummy

But, they are there on Linux (official Docker image):

echo $LD_LIBRARY_PATH

/home/openfoam/OpenFOAM/openfoam-v2306/platforms/linuxARM64GccDPInt32Opt/lib:/usr/lib/openfoam/openfoam2306/site/2306/platforms/linuxARM64GccDPInt32Opt/lib:/usr/lib/openfoam/openfoam2306/platforms/linuxARM64GccDPInt32Opt/lib/sys-openmpi:/usr/lib/openfoam/openfoam2306/platforms/linuxARM64GccDPInt32Opt/lib:/usr/lib/aarch64-linux-gnu/openmpi/lib:/usr/lib/openfoam/openfoam2306/platforms/linuxARM64GccDPInt32Opt/lib/dummy

Just found a bug in foamAddLib, which does not add these folders to FOAM_LD_LIBRARY_PATH.

I have added a folder existence check to foamAddLib function to avoid adding non-existing folders to FOAM_LD_LIBRARY_PATH. Unfortunately, I did not expect $FOAM_USER_LIBBIN:$FOAM_SITE_LIBBIN as an argument.

Just found a bug in foamAddLib, which does not add these folders to FOAM_LD_LIBRARY_PATH.

All right, but the consensus is that they should be added, right? Well, then I'm back to being torn on whether to set $FOAM_USER_LIBBIN as an rpath...

I'm still uncomfortable with silently storing a user-dependent path inside the binaries. It also won't do anything for my project, or for anyone who's not compiling just for their own use. The fact that this has been broken on macOS for a while and no one found it before makes me think that this feature might not be that important either.

On the other hand, I can see why someone would want the same behavior as on Linux, and right now I don't have a different suggestion on how to get that for this particular use case. So, I guess I'd be happy with a compilation flag that enables/disables this part of the patch (should ideally control only whether $FOAM_USER_LIBBIN is set as an rpath; the rest of the changes here are fully welcome by me!). Also, for the same reason, the current $FOAM_LD_LIBRARY_PATH workaround will unfortunately have to stay.

@alexey or @alexey - is there something special with SIP having problems with non-existent directories, or is this an overly-aggressive optimization?

I would think that having FOAM_USER_LIBBIN in the rpath is probably not particularly useful, since the original user-specific location is then baked into the binary so the FOAM_LD_LIBRARY_PATH or the @executable_path is probably the way to go there.

Not so completely certain about FOAM_SITE_LIBBIN - it could be useful for group-wide configurations, or should there be an additional ifneq (,$(wildcard $(FOAM_SITE_LIBBIN))) check as well?

In either case, I hope that these rule changes will bring relief for SIP problems.

@mark No, there is no problem with non-existent folders. It was an attempt to avoid a pollution of environment with non-existent folders (mainly third party folders).

We can drop FOAM_USER_LIBBIN, if a user really wants it in rpath, she can use FOAM_EXTRA_LD_FLAGS to pass additional flags to linker (i.e. FOAM_EXTRA_LD_FLAGS = -rpath $FOAM_USER_LIBBIN).

I am not sure if we need to check existence of FOAM_SITE_LIBBIN during compilation. In case of site customisations execution and compilation computers are different with high probability.

We can drop FOAM_USER_LIBBIN, if a user really wants it in rpath, she can use FOAM_EXTRA_LD_FLAGS to pass additional flags to linker (i.e. FOAM_EXTRA_LD_FLAGS = -rpath $FOAM_USER_LIBBIN).

@alexey If you're okay dropping $FOAM_USER_LIBBIN altogether, then that of course works for me! Sorry for feeling so strongly about this one.

Not so completely certain about FOAM_SITE_LIBBIN - it could be useful for group-wide configurations

@mark IMO the same reasoning of $FOAM_USER_LIBBIN applies to $FOAM_SITE_LIBBIN... it should be set as an rpath to allow a first-party (or, I guess, user) binary to load a site-installed library in cases where SIP is involved (i.e., SIP is enabled and you're running an Apple shell) but the $FOAM_LD_LIBRARY_PATH workaround can't kick in (e.g. you aren't using a run script that sources RunFunctions).

TBH I still think it's a minor edge case that might not be worth supporting through rpath entries. However, given that neither the user's name nor their $HOME are part of the resolved path by default, I don't feel nearly as strongly about this one.

or should there be an additional ifneq (,$(wildcard $(FOAM_SITE_LIBBIN))) check as well?

I am not sure if we need to check existence of FOAM_SITE_LIBBIN during compilation. In case of site customisations execution and compilation computers are different with high probability.

@mark @alexey I don't think a check here is needed nor desirable (the point of setting $FOAM_SITE_LIBBIN as an rpath is to be able to override libraries... the behavior here shouldn't depend on whether $FOAM_SITE_LIBBIN existed at the time OpenFOAM was built or it was created later).

I have tested solvers and libraries installed in $FOAM_USER_APPBIN and $FOAM_USER_LIBBIN, and it seems we won't be able to solve SIP problem without linking-in absolute path to $FOAM_LIBBIN (and $FOAM_LIBBIN/$FOAM_MPI with $FOAM_LIBBIN/dummy).

If we have an executable installed in $FOAM_USER_APPBIN, @executable_path/../lib points to $FOAM_USER_LIBBIN. So, upon execution the binary cannot find base OpenFOAM installation and libraries. Execution fails with the following error (I have simply copied icoFoam to $FOAM_USER_APPBIN):

dyld[12360]: Library not loaded: @rpath/libfiniteVolume.dylib
  Referenced from: <5D89D34C-4680-3E3B-AEFD-0676EFD52A0F> $FOAM_USER_APPBIN/icoFoam
  Reason: tried: $FOAM_USER_LIBBIN/libfiniteVolume.dylib, (no such file), ... '/usr/local/lib/libfiniteVolume.dylib' (no such file), '/usr/lib/libfiniteVolume.dylib' (no such file, not in dyld cache)

In case of a library @loader_path is resolved to $FOAM_USER_LIBBIN. And again dynamic loader cannot find OpenFOAM base. Which leads to library load failure (here I have compiled CourantNo function object as a separate library in $FOAM_USER_LIBBIN):

--> FOAM Warning : 
    From void *Foam::dlLibraryTable::openLibrary(const Foam::fileName &, bool)
    in file db/dynamicLibrary/dlLibraryTable/dlLibraryTable.C at line 188
    Could not load "libCourantNo"
dlopen(libCourantNo.dylib, 0x0009): symbol not found in flat namespace '__ZN4Foam15functionObjects15fieldExpression5clearEv'

We would expect the same behaviour for executables and libraries, installed in $FOAM_SITE_APPBIN/$FOAM_SITE_LIBBIN.

I have tested solvers and libraries installed in $FOAM_USER_APPBIN and $FOAM_USER_LIBBIN, and it seems we won't be able to solve SIP problem without linking-in absolute path to $FOAM_LIBBIN (and $FOAM_LIBBIN/$FOAM_MPI with $FOAM_LIBBIN/dummy).

Yes, these need to be set. Good catch.

I thought they were being set, but I can confirm via otool that they're not (the $FOAM_LD_LIBRARY_PATH workaround hides this problem). These probably belong after @loader_path/@executable_path/../lib, following how $[FOAM/DY]LD_LIBRARY_PATH works and to not interfere with relocatability of the OpenFOAM installation (assuming that first-party binaries will also have these rpaths, even if they're redundant there).

BTW, I see that some of the libraries in $FOAM_LIBBIN/$FOAM_MPI and $FOAM_LIBBIN/dummy are looking for $LIBBIN/libdecompositionMethods.dylib via @rpath/libdecompositionMethods.dylib. For that lookup to work, these libraries should get an extra rpath entry pointing to @loader_path/..!

Next iteration updated:

no user libbin (too fragile)
find mpi-specific before serial libraries
dummy stub location last

I didn't yet address the group (SITE) location, pending more feedback either way.

Not sure of a good way to handle things like libptscotchDecomp, which will generally land under the FOAM_LIBBIN/FOAM_MPI location and this not find libdecompositionMethods since it is located in the ../folder instead of the ./folder

Is it reasonable (although it looks pretty stupid) if we always included @loader_path/.. too? For the regular compilation we have this type of structure:

platforms/darwin64ClangDPInt32Opt/bin
platforms/darwin64ClangDPInt32Opt/lib
platforms/darwin64ClangDPInt32Opt/lib/sys-openmpi

so the @loader_path/.. (needed for the mpi-specific libs) would refer to platforms/darwin64ClangDPInt32Opt for serial libs. Not particularly bad, but a bit odd. It does however properly resolve to platforms/darwin64ClangDPInt32Opt/lib for mpi-specific libs.

For the mapping @loader_path/lib -> @executable_path/../lib, the extra @loader_path/.. hack would then resolve to so the @executable_path/../../lib or platforms/lib in this example.

Not sure where this is going...

@mark Thanks for the updated branch.

I didn't yet address the group (SITE) location, pending more feedback either way.

Considering my particularly strong opinion on $FOAM_USER_LIBBIN, I'd prefer to leave this one for @alexey and you to decide.

Is it reasonable (although it looks pretty stupid) if we always included @loader_path/.. too? For the regular compilation we have this type of structure:

It does look stupid yet it should work. TBH I don't particularly like it, but I can't quickly come up with an alternative that doesn't require special rules or resorting to symlinks (both of which I assume we're trying to avoid). If @alexey has an opinion on how to better deal with this, then I'd say we along with that; otherwise I believe a @loader_path/.. rpath entry that appears after @loader_path isn't that bad.

Just found a bug in foamAddLib, which does not add these folders to FOAM_LD_LIBRARY_PATH.

I have added a folder existence check to foamAddLib function to avoid adding non-existing folders to FOAM_LD_LIBRARY_PATH. Unfortunately, I did not expect $FOAM_USER_LIBBIN:$FOAM_SITE_LIBBIN as an argument.

@alexey I've just gotten a bug report in my project that seems to be caused by this -- is this bug being tracked anywhere and/or fixed already?

In v2312 it is still there:

_foamAddLib()
{
    case "$1" in (/?*)
    if [ -e "$1" ]
    then
        export FOAM_LD_LIBRARY_PATH="${1}${FOAM_LD_LIBRARY_PATH:+:}${FOAM_LD_LIBRARY_PATH}"
        export DYLD_LIBRARY_PATH="$FOAM_LD_LIBRARY_PATH"
    fi
    esac
}

Guess, I need to submit a fix for sh and csh functions file.

No need for a new issue or patch, I can easily patch it with this same issue number. Eg,

-# OpenFOAM user, group libraries
-_foamAddLib  "${FOAM_USER_LIBBIN}:${FOAM_SITE_LIBBIN}"
+_foamAddLib  "$FOAM_SITE_LIBBIN"  # OpenFOAM group libraries
+_foamAddLib  "$FOAM_USER_LIBBIN"  # OpenFOAM user  libraries

However, I noticed that the csh/functions has this:

# Special treatment for Darwin
# - DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH
if ("${_foam_uname_s}" == 'Darwin') then
    alias _foamAddLib 'if (-e \!*) setenv DYLD_LIBRARY_PATH \!*\:${DYLD_LIBRARY_PATH}; if (-e \!*) setenv FOAM_LD_LIBRARY_PATH \!*\:${FOAM_LD_LIBRARY_PATH}'
else
    alias _foamAddLib 'setenv LD_LIBRARY_PATH \!*\:${LD_LIBRARY_PATH}'
endif

Could either of you verify if that can be simplified? For example,

# Special treatment for Darwin
# - DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH
if ("${_foam_uname_s}" == 'Darwin') then
    alias _foamAddLib 'if (-e \!*) then; setenv FOAM_LD_LIBRARY_PATH \!*\:${FOAM_LD_LIBRARY_PATH}; setenv DYLD_LIBRARY_PATH ${FOAM_LD_LIBRARY_PATH}; endif'
else
    alias _foamAddLib 'setenv LD_LIBRARY_PATH \!*\:${LD_LIBRARY_PATH}'
endif

In fact, I do not know if the idea of checking of folder existence (-e) is really good idea.

Initially I have added this check to avoid polluting environment variables with missing paths. Now I think that this check should be removed, as third-party library folders could be mounted later.

That was also my initial thought when I saw it, but will go with whatever you think.

If we are to remove that check, do make a new issue.

Created issue #3098 (closed).

otool -l /Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib/libOpenFOAM.dylib

/Volumes/OpenFOAM-rpath/platforms/darwin64ClangDPInt32Opt/lib/libOpenFOAM.dylib:
[[[ NOTE: lines removed for brevity ]]]
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libz.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 21:00:02 1969
      current version 1.2.11
compatibility version 1.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libPstream.dylib (offset 24)
   time stamp 2 Wed Dec 31 21:00:02 1969
      current version 0.0.0
compatibility version 0.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 21:00:02 1969
      current version 1300.36.0
compatibility version 1.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Wed Dec 31 21:00:02 1969
      current version 1319.0.0
compatibility version 1.0.0
Load command 16
          cmd LC_RPATH
      cmdsize 88
         path /Users/gabriel/OpenFOAM/gabriel-com/platforms/darwin64ClangDPInt32Opt/lib (offset 12)
Load command 17
          cmd LC_RPATH
      cmdsize 88
         path /Volumes/OpenFOAM-rpath/site/2306/platforms/darwin64ClangDPInt32Opt/lib (offset 12)
Load command 18
          cmd LC_RPATH
      cmdsize 32
         path @loader_path (offset 12)
Load command 19
          cmd LC_RPATH
      cmdsize 32
         path @loader_path/dummy (offset 12)
Load command 20
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 6905992
 datasize 28600
Load command 21
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 6934592
 datasize 0
Load command 22
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 9512720
 datasize 74462

Just in case, to verify:

echo $FOAM_MPI

openmpi

mentioned in commit fb2b3a6f

added community contribution labels

assigned to @mark

unassigned @mark

assigned to @mark

mentioned in commit 77675a24

mentioned in issue #2956 (closed)

mentioned in commit d679e46a

mentioned in commit e4f2efec

Added a reduced form that mostly relies on @loader_path (see comments on commit) but still leaves the shadow env variable. Still not really a complete solution I guess, but a step in the right direction. Thank you for the code contribution and (to all) for the good and timely interactions and testing. Added to develop - closing the issue for now.

closed

mentioned in commit 852f66fc

Hello,

Sorry to bring up this relatively old issue, but my problem might be related.

I added some extra CXXFLAGS and built OpenFOAM-v2412 in debug-mode (since WM_COMPILE_OPTION=Debug had no effect I had to add additional flags). After compiling solids4foam, I tried to debug it using lldb, but it failed to find libOpenFOAM.dylib.

$ lldb solids4Foam
(lldb) target create "solids4Foam"
Current executable set to '/Users/ali/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/bin/solids4Foam' (arm64).
(lldb) b main
Breakpoint 1: where = solids4Foam`main + 44 at setRootCase.H:5:15, address = 0x0000000100001048
(lldb) r
Process 26479 launched: '/Users/ali/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/bin/solids4Foam' (arm64)
dyld[26479]: Library not loaded: @rpath/libOpenFOAM.dylib
  Referenced from: <AF51F016-8EF7-4163-83FA-7746AFBF582D> /Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/bin/solids4Foam
  Reason: tried: '/Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/lib/sys-openmpi/libOpenFOAM.dylib' (no such file), '/Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/lib/libOpenFOAM.dylib' (no such file), '/Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/lib/dummy/libOpenFOAM.dylib' (no such file), '/Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/lib/sys-openmpi/libOpenFOAM.dylib' (no such file), '/Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/lib/libOpenFOAM.dylib' (no such file), '/Volumes/code/opt/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/lib/dummy/libOpenFOAM.dylib' (no such file)
Process 26479 stopped

The issue was that lldb was search for it in $FOAM_USER_LIBBIN. I inspected solids4Foam with otool and here's some output:

$ otool -L $FOAM_USER_APPBIN/solids4Foam
/Users/ali/OpenFOAM/ali-v2412/platforms/darwin64ClangDPInt64Debug/bin/solids4Foam:
        @rpath/libsolids4FoamModels.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libOpenFOAM.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1800.105.0)

$ otool -l $FOAM_USER_APPBIN/solids4Foam | grep -A2 LC_RPATH
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../lib/sys-openmpi (offset 12)
--
          cmd LC_RPATH
      cmdsize 40
         path @executable_path/../lib (offset 12)
--
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../lib/dummy (offset 12)

Which confirms this issue.

My quick fix was to manually add $FOAM_LIBBIN as an rpath to solids4Foam,

$ install_name_tool -add_rpath $FOAM_LIBBIN $FOAM_USER_APPBIN/solids4Foam

$ otool -l $FOAM_USER_APPBIN/solids4Foam | grep -A2 LC_RPATH
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../lib/sys-openmpi (offset 12)
--
          cmd LC_RPATH
      cmdsize 40
         path @executable_path/../lib (offset 12)
--
          cmd LC_RPATH
      cmdsize 48
         path @executable_path/../lib/dummy (offset 12)
--
          cmd LC_RPATH
      cmdsize 88
         path /Users/ali/OpenFOAM/OpenFOAM-v2412/platforms/darwin64ClangDPInt64Debug/lib (offset 12)

OS: macos sonoma 14.7.1
API: 2402
$WM_OPTIONS: darwin64ClangDPInt64Debug
$FOAM_EXTRA_CXXFLAGS: -g3 -glldb -O0 -DFULLDEBUG -fno-omit-frame-pointer -fdebug-macro
$FOAM_EXTRA_CFLAGS: -g3 -glldb -O1 -DFULLDEBUG -fno-omit-frame-pointer -fdebug-macro
Note: I used this M1.patch.

Hi @reverseila, Without wanting to sound to brusque about it, we need to be clear about what we want to achieve and what is reasonably achievable.

The purpose of the rpath addition is to allow running of applications without additional setting of the LD_LIBRARY_PATH (DYLD_LIBRARY_PATH). If things are compiled into the regular OpenFOAM locations (FOAM_APPBIN, FOAM_LIBBIN) then the rpath logic of searching @executable_path/../lib seems to work it appears. If, however, we want to compile an executable into some other arbitrary location (eg, FOAM_USER_APPBIN), this logic obviously falls apart since there is no connection between the executable's location and the required libraries. We could go brute force on it and always add the absolute path to FOAM_LIBBIN (and MPI bits) to all executables. But if you do that OpenFOAM becomes non-relocatable.

So either you either need to set the environment variables, compile your application (or move the executable) into the FOAM_APPBIN or make the rpath setting non-relocatable (for everyone). If we follow the strictness of MacOS, then we could say that anything that is not directly within the OpenFOAM tree is not actually part of the application.

Thank you @mark for your clarification. That totally makes sense now. But, what environment variables should I set? DYLD_LIBRARY_PATH already contains FOAM_LIBBIN.

I hope someone in this chain with more MacOS experience can help.

@reverseila Usually lldb starts with empty environment. Did you try launching the process and then attaching to it by PID?

@alexey, thank you for your hint. However, the program's lifespan is short and ends before I get the chance to attach to it.

I came across this stackoverflow answer, which attributes the issue to macos' System Integrity Protection. I'd previously installed llvm via homebrew but hadn't add its binaries to PATH. llvm's lldb worked even when I deleted the rpath from the binary.

rpath compilation rules for Darwin

Functionality to add/problem to solve

Target audience

Proposal

What does success look like, and how can we measure that?

Links / references

Child items

Activity

Admin message

rpath compilation rules for Darwin

Functionality to add/problem to solve

Target audience

Proposal

What does success look like, and how can we measure that?

Links / references

Child items

Linked items

Activity