Reimplementation of dlOpen for macOS to avoid SIP problems

What does the apple fork() do versus vfork()? I'm a bit scared of making that change - see #185 (closed) - until I know more about what their fork does under the hood.

mentioned in commit f584ec97

closed

Hi @alexey - merged in your changes (in a few commits, with some modification).

I think that your changes look quite reasonable for dynamic loading (eg, dynamic code compilation etc), but what happens to all of the other entries in DYLD_LIBRARY_PATH such as third-party scotch/metis libraries. If I understand it correctly, SIP will clear out things so I'm not sure how these libraries get found properly.

BTW: I dropped out your vfork() to fork() as mentioned above.

Thanks for the merge. I think Scotch and other third-party libraries will be found as they are linked with full path, i.e.:

% otool -L libscotchDecomp.dylib
/Volumes/OpenFOAM/OpenFOAM-vdev/platforms/darwin64ClangDPInt32Opt/lib/libscotchDecomp.dylib (compatibility version 0.0.0, current version 0.0.0)
	/Volumes/OpenFOAM/OpenFOAM-vdev/platforms/darwin64ClangDPInt32Opt/lib/libdecompositionMethods.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/scotch/lib/libscotch.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/local/opt/scotch/lib/libscotcherrexit.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

Here I use system-wide installed Scotch.

Concerning fork/vfork issue I can only cite manual pages. On macOS:

NAME
     vfork – deprecated system call to create a new process

SYNOPSIS
     #include <unistd.h>

     pid_t
     vfork(void);

DESCRIPTION
     The vfork system call can be used to create new processes. As of macOS 12.0, this system call behaves identically to the
     fork(2) system call, except without calling any handlers registered with pthread_atfork(2).

     This system call is deprecated. In a future release, it may begin to return errors in all cases, or may be removed entirely.
     It is extremely strongly recommended to replace all uses with fork(2) or, ideally, posix_spawn(3).

On Linux everything is a bit different:

...
Linux description
       vfork(), just like fork(2), creates a child process of the
       calling process.  For details and return value and errors, see
       fork(2).

       vfork() is a special case of clone(2).  It is used to create new
       processes without copying the page tables of the parent process.
       It may be useful in performance-sensitive applications where a
       child is created which then immediately issues an execve(2).
...
CONFORMING TO

       4.3BSD; POSIX.1-2001 (but marked OBSOLETE).  POSIX.1-2008 removes
       the specification of vfork().
...

Though I doubt there are macOS users with InfiniBand, I will not insist on the change. Let's wait for the release, when it is removed, and then look for the solution.

added community label

FYI, I'm seeing a regression with the develop branch on macOS (both x86_64 and arm64), where some common instances (e.g. running the Allrun-parallel script of tutorials/basic/laplacianFoam/flange in the default macOS shell) in which the logic in RunFunctions would have successfully worked around the SIP issue have reverted to causing the previously seen error, i.e.:

--> FOAM FATAL ERROR: (openfoam-2306)
The dummy Pstream library cannot be used in parallel mode


    From static bool Foam::UPstream::init(int &, char **&, const bool)
    in file UPstream.C at line 49.

FOAM exiting

I can only suggest adding back the removed lines at https://develop.openfoam.com/Development/openfoam/-/blob/4284d02c996e2f9c281e1fe806063a7552332ac4/bin/tools/RunFunctions#L28-32 before the new release if at all possible, as that seems to undo the regression at least with the case I mentioned.

I confirm the behaviour. With empty DYLD_LIBRARY_PATH executables load dummy versions of libraries (Pstream, scotchDecomp, etc).

Reverting RunFunctions modification resolves the problem.

assigned to @mark

This stuff is really becoming a nightmare - glad however that this was caught before release. I'll reinstate the workarounds (RunFunctions and foamJob), but would like to know how other applications address this.

mentioned in commit b264d2e9

but would like to know how other applications address this.

My two cents...

But first, a small disclaimer: regarding the SIP issue I'm quite content with the previous workaround (i.e., that of #2555 (closed)). That, coupled with a non-system (i.e., user-installed) shell made it so that I haven't seen any problems nor had any issue reports on my project related to SIP. I agree with the objective set for what was proposed here, but I haven't really looked at the actual changes nor their implications besides the minimum for tracing the regression I saw.

I see it as, Apple would like users/developers to never use $DYLD_LIBRARY_PATH at all. The simplest alternative to that would be to encode the actual loader paths in the binaries.

E.g., let's take libOpenFOAM on macOS and look at what it's linked against:

otool -L /Volumes/OpenFOAM-develop/platforms/darwin64ClangDPInt32Opt/lib/libOpenFOAM.dylib 
/Volumes/OpenFOAM-develop/platforms/darwin64ClangDPInt32Opt/lib/libOpenFOAM.dylib:
	/Volumes/OpenFOAM-develop/platforms/darwin64ClangDPInt32Opt/lib/libOpenFOAM.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/Volumes/OpenFOAM-develop/platforms/darwin64ClangDPInt32Opt/lib/dummy/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

Output shows this lib is looking for OpenFOAM's dummy variant of libPstream. This is in principle just fine for OpenFOAM because it just sets $(DY)LD_LIBRARY_PATH to the correct dir containing the "real" variant of that lib. But, Apple seems to not want $DYLD_LIBRARY_PATH to be used to override which library is loaded (at least, not across different shells/executables; the rationale is that it's bad for security), so the contents of $DYLD_LIBRARY_PATH will be cleared between processes in many instances.

This however wouldn't be a problem if (at least on macOS), the library load paths didn't need to be overridden via $DYLD_LIBRARY_PATH at all and libOpenFOAM were directly linked against the real libPstream.dylib instead of the dummy one. This could be done as it's compiled; or, the binary can be "fixed" after compilation using install_name_tool. I've experimented with fixing libOpenFOAM with install_name_tool after compilation so that it looks for the real libPstream, and that seemed to work. I can only guess that this is what Apple would prefer devs did.

EDIT: typos

BTW, how necessary is the use of separate "dummy" and "real" libs? I believe OpenFOAM does it so that the user can switch between MPI implementations without recompilation (is this correct?). Yet, the dummy lib is expected to fail whenever it's used, so maybe the build process could potentially be changed so that the dummy isn't built at all, and OpenFOAM has a link against a "real" library by default?

Main difficulty with Allrun scripts and SIP is usage of /bin/sh. Since /bin/sh is a part of system, it is restricted and [1]:

Spawning children processes of processes restricted by System Integrity
Protection, such as by launching a helper process in a bundle with
NSTask or calling the exec(2) command, resets the Mach special ports
of that child process. Any dynamic linker (dyld) environment variables,
such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

As you have noted, switching to a shell, installed in /usr/local/bin removes the problem. Because /usr/local is excluded from SIP.

Concerning dummy/real: when solvers/utilities are executed in serial mode, dummy libraries are used (since there is no need in communications). In parallel mode MPI-enables libraries are loaded. This is achieved by correctly constructing DYLD_LIBRARY_PATH variable. If we link executables with MPI-enabled libraries, we won't be able to run executables in serial mode, as they start to fail with error:

    if (worldIndex == -1 && numprocs <= 1)
    {
        FatalErrorInFunction
            << "attempt to run parallel on 1 processor"
            << Foam::abort(FatalError);
    }

One of the solutions to remedy empty DYLD_LIBRARY_PATH, that I saw, is to use RPATH. Unfortunately, I did not have time to test it yet.

As you have noted, switching to a shell, installed in /usr/local/bin removes the problem. Because /usr/local is excluded from SIP.

Yeah, this helps a lot when combined with the FOAM_LD_LIBRARY_PATH workaround. As you've said before, it doesn't fix all cases (you need to source the RunFunctions in any script you invoke for the workaround to have effect); but I think it already gets us most of the way there.

Concerning dummy/real: when solvers/utilities are executed in serial mode, dummy libraries are used (since there is no need in communications). In parallel mode MPI-enables libraries are loaded. This is achieved by correctly constructing DYLD_LIBRARY_PATH variable.

Not saying this is not right (I haven't looked at the code for that to see how it works), but that hasn't been my experience with it, as I've previously replaced the install names in the binaries as a fix for the SIP problem with success, and that seemed to work both for serial runs and in parallel. That has to mean that the MPI-enabled lib can handle a serial run... or maybe I'm misunderstanding what you're referring to?

One of the solutions to remedy empty DYLD_LIBRARY_PATH, that I saw, is to use RPATH. Unfortunately, I did not have time to test it yet.

Does the rpath functionality have any feature that would allow for runtime/dynamic selection of what to load? AFAIK, the rpaths are stored directly in the binary (e.g. with install_name_tool), so it wouldn't be a straight replacement for DYLD_LIBRARY_PATH (but maybe I'm missing something). Though I think using rpaths can be a good way to add relocatability to the OpenFOAM installation.

For clarification (@gerlero and @alexey): can generally consider the dummy libraries as stubs. In the case of Pstream, this not only means that we can swap out MPI vendor libraries by simply recompiling src/Pstream and src/parallel/decompose (they correspondingly have an Allwmake-mpi script), which only takes about a minute. It also means that the dummy library provides a concrete symbol for link resolution (need by the clang lld linker and mingw).

In a serial simulation, the Pstream::init() is never called so it won't matter if you have a real or dummy MPI connection. If you do try to call Pstream::init() with a single rank, the dummy version will complain about being used at all, the MPI version will first call MPI_Initialize, get the number of ranks and then complain that about trying to use MPI with a single rank.

It could be that the rpath renaming is the way to solve some of the MacOS issues, but would prevent quick swapping of MPI backends. Can't really start replacing /bin/sh with /usr/local/bin/sh I think (too much scripting and not exactly portable either).

Thanks for the clarifications.

It could be that the rpath renaming is the way to solve some of the MacOS issues, but would prevent quick swapping of MPI backends.

Yeah, I think fixing the install names to always point to a valid backend can be a viable fix (and more robust than the current workaround). As I've mentioned, this seemed to work just fine when I tried it.

I also don't think many Mac users need to switch between MPI backends (and they still could, within the bounds of the current workarounds; this would just change which library is loaded by default when SIP gets in the way)

I might try adding such a step to my project. If that works, maybe we can then figure out a way to upstream it here.

If you do try to call Pstream::init() with a single rank, the dummy version will complain about being used at all, the MPI version will first call MPI_Initialize, get the number of ranks and then complain that about trying to use MPI with a single rank.

I understand then that it's still okay to be linked against an MPI-enabled backend when running serially (Pstream::init() would just never be called, but everything else would work just as if you were linked against the dummy backend). Right?

Correct - it would only be a problem if you had the MPI-specific Pstream linked in but did not have the corresponding MPI libraries actually available. In this case it would probably give a run-time link error for the various MPI symbols.

I might try adding such a step to my project. If that works, maybe we can then figure out a way to upstream it here.

One consideration may be that OpenFOAM currently does not really have a compile/install concept. It is essentially "installed" wherever you happened to last compile it or copy it. We do have the bin/tools/install-dirs and bin/tools/install-platform utilities, which were added mostly for generating RPMs. Could think about adding in an rpath hook there and then using them to install into particular locations for MacOS.

Can say I'm no longer catching a regression with the new change at b264d2e9. Thanks to both for the quick confirmation and fix.

As for my install_name_tool idea, I'm trying it out at https://github.com/gerlero/openfoam-app/pull/166.

This the script that makes the changes. I don't think it'll fix all problems related to SIP (I'm not sure this will help when using dynamicCode for example), but it should prevent some common errors.

Hi @gerlero - I haven't had time to examine a detailed integration of your script changes. Might have to make a few more iterations as well.

In either case, I don't actually have access to MacOS (in any flavour). Do you or @alexey have some updated notes for building OpenFOAM there? Something that we could use to update the wiki information.

Hi @gerlero - I haven't had time to examine a detailed integration of your script changes. Might have to make a few more iterations as well.

@mark The install-name patching is live at my project (https://github.com/gerlero/openfoam-app/blob/main/fix_install_names.sh). It hasn't caused any issues so far. I haven't really looked at the bin/tools/install-dirs/bin/tools/install-platform utilities you mention either, so I have no idea how it would be upstreamed it.

In either case, I don't actually have access to MacOS (in any flavour). Do you or @alexey have some updated notes for building OpenFOAM there? Something that we could use to update the wiki information.

My project (https://github.com/gerlero/openfoam-app) does not offer any text-based instructions, but it solves the problem of building on macOS by automating the process (as well as providing precompiled builds). I'm okay if you want to point macOS users at my project from that page.

@mark my idea of using run time path list requires changes in make rules. Basically, I will add -rpath option during link phase, where I add three paths: $FOAM_LIBBIN, $FOAM_LIBBIN/$FOAM_MPI, and $FOAM_LIBBIN/dummy.

Right now I have these additions to usual rules:

...
rpathFLAGS = -rpath $(FOAM_LIBBIN)
ifneq ($(FOAM_MPI),)
rpathFLAGS +=  -rpath $(FOAM_LIBBIN)/$(FOAM_MPI)
endif
rpathFLAGS += -rpath $(FOAM_LIBBIN)/dummy
...
LINKLIBSO   = $(CC) $(c++FLAGS) $(rpathFLAGS) -dynamiclib -install_name @rpath/$(notdir $(LIB)$(EXT_SO)) -Wl,-dylib,-undefined,dynamic_lookup

LINKEXE     = $(CC) $(c++FLAGS) $(rpathFLAGS) -Wl,-execute,-undefined,dynamic_lookup

This way we still can use DYDL_LIBRARY_PATH to change library lookup paths, but when it is empty, dynamic loader will fall-back to rpath mechanism for search.

I am testing new rules and it seems they do not break anything yet. I am not quite happy with absolute values for -rpath flags, I plan to change them to relative using @executable_path and @loader_path dynamic loaded variables, so, installation can be easily moved to a new location.

As soon as I finish testing, I will submit feature request.

I was wondering about using the relative rpath, since these things should generally be relocatable, but I thought this also meant needing to adhere to a strict ../bin, ../lib type of structure as well - or am I wrong there? Not a showstopper, but just curious.

For libraries rpath list should be:

$FOAM_USER_LIBBIN
$FOAM_SITE_LIBBIN
@loader_path
@loader_path/$FOAM_MPI
@loader_path/dummy

Libraries are either in $FOAM_USER_LIBBIN, $FOAM_SITE_LIBBIN, or $FOAM_LIBBIN (which is @loader_path in the case of a dynamic library). So, we have to keep dynamic libraries, which are looked up by dynamic loader, in these folders. Libraries, which are loaded by a name with dlOpen, should be located in one of the folders in DYLD_LIBRARY_PATH (or FOAM_LD_LIBRARY_PATH). Finally, there are libraries, which are loaded by full path (ex. results of codeStream compilation), they can be located anywhere.

For executables rpath list should be:

$FOAM_USER_LIBBIN
$FOAM_SITE_LIBBIN
@executable_path/../lib
@executable_path/../lib/$FOAM_MPI
@executable_path/../lib/dummy

In this case bin and lib folders should be on the same level in directory tree. And this is the only restriction of these relative run-time paths.

Probably have to drop FOAM_USER_LIBBIN from the list. Unless it is a very personal installation, everyone on the system will have a different user-libbin.

@mark my idea of using run time path list requires changes in make rules. Basically, I will add -rpath option during link phase, where I add three paths: $FOAM_LIBBIN, $FOAM_LIBBIN/$FOAM_MPI, and $FOAM_LIBBIN/dummy.

@alexey Thanks. I really think this is the way to go.

I am testing new rules and it seems they do not break anything yet. I am not quite happy with absolute values for -rpath flags, I plan to change them to relative using @executable_path and @loader_path dynamic loaded variables, so, installation can be easily moved to a new location.

Agreed. All entries pointing to the OpenFOAM install should be set to a relative (i.e., @loader_path-based) value to allow for relocation.

I was wondering about using the relative rpath, since these things should generally be relocatable, but I thought this also meant needing to adhere to a strict ../bin, ../lib type of structure as well - or am I wrong there? Not a showstopper, but just curious.

@mark, yeah, you can move the tree but you have to preserve the internal file structure. Also, dynamic libraries retain their install_names, so compiling new binaries against the relocated libraries might no longer work without manually fixing it... I haven't found a fix for that yet. It's still an improvement over using absolute paths.

FWIW (@alexey might be interested too), I've recently sent a pull request to Homebrew (https://github.com/Homebrew/brew/pull/15571) implementing this same idea for the Homebrew install.

For libraries rpath list should be: For executables rpath list should be:

@alexey Agree with @mark that $FOAM_USER_LIBBIN should be dropped (it cannot really be resolved at compile time).

Not sure if I'd keep $FOAM_SITE_LIBBIN either; this one can be resolved to a sensible path (although I guess such path could be later changed), but I don't expect OpenFOAM binaries to link agains sitebinaries. In any case, if it's kept, I'd put it last on the list.

@alexey Also, maybe this takes considerable extra effort and isn't really worth it at this stage, but ideally I'd like to see third-party libs handled the same way if they're located somewhere relative to the OpenFOAM install (i.e., if they're not installed at the system level).

E.g. in my case I have:

echo $DYLD_LIBRARY_PATH

/Volumes/OpenFOAM-v2306/platforms/darwin64ClangDPInt32Opt/lib/openmpi:/Volumes/OpenFOAM-v2306/platforms/darwin64ClangDPInt32Opt/lib:/Volumes/OpenFOAM-v2306/usr/opt/fftw/lib:/Volumes/OpenFOAM-v2306/usr/opt/mpfr/lib:/Volumes/OpenFOAM-v2306/usr/opt/gmp/lib:/Volumes/OpenFOAM-v2306/usr/opt/cgal/lib:/Volumes/OpenFOAM-v2306/usr/opt/boost/lib:/Volumes/OpenFOAM-v2306/usr/opt/open-mpi/lib:/Volumes/OpenFOAM-v2306/platforms/darwin64ClangDPInt32Opt/lib/dummy

so it'd be ideal if installed OpenFOAM binaries can resolve all those paths via relative references. This would allow relocation of an install with third-party dependencies inside (as long as those third-party dependencies are also relocatable themselves).

@gerlero @mark I have created an issue #2948 (closed), where I have submitted new compilation rules for Darwin.

I think FOAM_USER_LIBBIN is relevant, since people usually build software manually on macOS, so this location is known at the moment of compilation.

I am not sure if it is worth messing with manual compilation of third party libraries. Some of them need special patches to compile with clang, some of them can only be compiled with gcc. And all third party build scripts (automake, cmake, custom scripts) has to be adapted to this approach. Theoretically we can edit install names of the compiled libraries with install_name_tool, yet, won't it be easier to symlink all third-party libraries into $FOAM_LIBBIN/third-party and add this folder to -rpath flag?

Thanks. I'll reply there.

Reimplementation of dlOpen for macOS to avoid SIP problems

Functionality to add/problem to solve

Target audience

Proposal

What does success look like, and how can we measure that?

Funding

Note

Child items

Activity

Admin message

Reimplementation of dlOpen for macOS to avoid SIP problems

Functionality to add/problem to solve

Target audience

Proposal

What does success look like, and how can we measure that?

Funding

Note

Child items

Linked items

Activity