Dynamic library search fails on macOS if solver is executed using foamJob script

Summary

Dynamic library search fails on macOS if solver is executed using foamJob script because SIP clears DYLD_LIBRARY_PATH variable.

Steps to reproduce

Execute any case, which loads additional dynamic libraries, using foamJob script.

Example case

initChannel case from tutorials/incompressible/pimpleFoam/LES/surfaceMountedCube. In Allrun script I have changed runApplication simpleFoam to foamJob simpleFoam.

What is the current bug behaviour?

Upon execution solver prints warning about missing libraries:

--> FOAM Warning : 
    From void *Foam::dlLibraryTable::openLibrary(const Foam::fileName &, bool)
    in file db/dynamicLibrary/dlLibraryTable/dlLibraryTable.C at line 188
    Could not load "fieldFunctionObjects"
dlopen(libfieldFunctionObjects.dylib, 0x0009): tried: 'libfieldFunctionObjects.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibfieldFunctionObjects.dylib' (no such file), '/usr/lib/libfieldFunctionObjects.dylib' (no such file, not in dyld cache), 'libfieldFunctionObjects.dylib' (no such file), '/usr/local/lib/libfieldFunctionObjects.dylib' (no such file), '/usr/lib/libfieldFunctionObjects.dylib' (no such file, not in dyld cache)
--> FOAM Warning : 
Unknown function type turbulenceFields

What is the expected correct behavior?

Solver finds requested libraries and loads them.

Relevant logs and/or images

See attached log.gz.

Environment information

• OpenFOAM version : v2206 and vdev (868d6dd7)
• Operating system : macOS 13.4
• Compiler : Apple clang version 14.0.3 (clang-1403.0.22.14.1)

Possible fixes

Restore DYLD_LIBRARY_PATH from FOAM_DYLD_LIBRARY_PATH, like in RunFunctions:

# Darwin workaround - SIP clearing DYLD_LIBRARY_PATH variable
if [ -n "$FOAM_DYLD_LIBRARY_PATH" ] && [ -z "$DYLD_LIBRARY_PATH" ]
then
    export DYLD_LIBRARY_PATH="$FOAM_DYLD_LIBRARY_PATH"
fi

@mark @Pawan

added bug community labels

assigned to @mark

Making the suggest change. I start to wonder we need to bury in the handling for FOAM_LD_LIBRARY_PATH within the source code itself? It is all a bit messy with SIP.

We can try to emulate dydl behaviour: iterate through paths in FOAM_LD_LIBRARY_PATH and load library by full path instead of a name. This PoC seems to do the trick (at least it solves the problem, reported here):

#if defined(__APPLE__)
#define FOAM_DLOPEN foamDlopen

void *foamDlopen(const char* path, int mode)
{
	const std::string dyldPathVariable("DYLD_LIBRARY_PATH");
	std::string varVal(Foam::getEnv(dyldPathVariable));
	if (!varVal.empty())
	{
		if (Foam::POSIX::debug)
		{
			std::cout << "Using shortcut for non-empty " << dyldPathVariable << "." << std::endl;
		}

		return ::dlopen(path, mode);
	}

	const std::string backupDyldPathVariable("FOAM_LD_LIBRARY_PATH");

	if (Foam::POSIX::debug)
	{
		std::cout << "Iterating through paths in " << backupDyldPathVariable << "." << std::endl;
	}

	std::string backupVal(Foam::getEnv(backupDyldPathVariable));
	auto paths = Foam::stringOps::split(backupVal, ':');
	for(const auto& pth : paths)
	{
		Foam::fileName p(Foam::fileName(pth)/Foam::fileName(path));
		void* hdl = ::dlopen(p.c_str(), mode);
		if (hdl != nullptr)
			return hdl;
	}
	return nullptr;
}
#else
#define FOAM_DLOPEN ::dlopen
#endif

void* Foam::dlOpen(const fileName& libName, const bool check)
{
    constexpr int ldflags = (RTLD_LAZY|RTLD_GLOBAL);


    if (POSIX::debug)
    {
        std::cout
            << "dlOpen(const fileName&)"
            << " : dlopen of " << libName << std::endl;
    }

    void* handle = FOAM_DLOPEN(libName.c_str(), ldflags);

    if (!handle)
    {
        fileName libso;

        if (!libName.has_path() && !libName.starts_with("lib"))
        {
            // Try with 'lib' prefix
            libso = "lib" + libName;
            handle = FOAM_DLOPEN(libso.c_str(), ldflags);

            if (POSIX::debug)
            {
                std::cout
                    << "dlOpen(const fileName&)"
                    << " : dlopen of " << libso << std::endl;
            }
        }
        else
        {
            libso = libName;
        }

        // With canonical library extension ("so" or "dylib"), which remaps
        // "libXX" to "libXX.so" as well as "libXX.so" -> "libXX.dylib"
        if (!handle && !libso.has_ext(EXT_SO))
        {
            libso.replace_ext(EXT_SO);
            handle = FOAM_DLOPEN(libso.c_str(), ldflags);

            if (POSIX::debug)
            {
                std::cout
                    << "dlOpen(const fileName&)"
                    << " : dlopen of " << libso << std::endl;
            }
        }
    }

    if (!handle && check)
    {
        WarningInFunction
            << "dlopen error : " << ::dlerror() << endl;
    }

    if (POSIX::debug)
    {
        std::cout
            << "dlOpen(const fileName&)"
            << " : dlopen of " << libName
            << " handle " << handle << std::endl;
    }

    return handle;
}

If it seems reasonable, I can proceed with a feature request submission as a separate issue.

This look reasonable. For style, would prefer a local function instead of a define. Ie something like this:

namespace Foam
{

static void* open_library(const Foam::string& libName)
{
    constexpr int ldflags = (RTLD_LAZY|RTLD_GLOBAL);

    #ifdef __APPLE__
    // Apple SIP tends to clear out DYLD_LIBRARY_PATH

    const char* env = ::getenv("DYLD_LIBRARY_PATH");
    if (env && strlen(env) != 0)
    {
        return ::dlopen(libName, ldflags);
    }
    
    // Use FOAM_LD_LIBRARY_PATH fallback
    env = ::getenv("FOAM_LD_LIBRARY_PATH");
    if (env)
    {
        const Foam::string fallbackPaths(env);
	const auto paths = Foam::stringOps::split<Foam::string>(fallbackPaths, ':');

        ... more stuff
	for (const auto& root : paths)
	{
   ... need to check if we require str() method on the substrings ....
            Foam::fileName fullPath(root.str()/libName);
            void* handle = ::dlopen(fullPath.c_str(), ldflags);
	    if (handle)
            {
	        return handle;
            }
	}

    }
    #else
    // Normal handling. Simply use dlopen
    handle = ::dlopen(libName.c_str(), ldflags);
    #endif

    return handle;
}

} // End namespace

Should be low-impact enough (APPLE only) that it could make it for 2306.

Would it work to simply force the env variable ourselves? Or does the system block that?

Eg,

{
    #ifdef __APPLE__
    // Apple SIP tends to clear out DYLD_LIBRARY_PATH

    const char* env = ::getenv("DYLD_LIBRARY_PATH");
    if (!env || !strlen(env))
    {
        // Use FOAM_LD_LIBRARY_PATH fallback
        env = ::getenv("FOAM_LD_LIBRARY_PATH");

        if (env || strlen(env) != 0)
        {
            ::setenv("DYLD_LIBRARY_PATH", env, true);  // overwrite
        }
    }
    #endif

    return ::dlopen(libName.c_str(), ldflags);
}

Could perhaps even hack that into argList?

In fact, this approach was my first solution, and it does not work. Process takes into account state of the variable (DYDL_LIBRARY_PATH) at the beginning of the execution.

Thought so. Computers are so smart...

mentioned in commit 57c50bda

thanks

closed

mentioned in issue #2801 (closed)