Skip to content

Introducing MUI code coupling library into OpenFOAM

Functionality to add/problem to solve

Integrate the code coupling library Multiscale Universal Interface in OpenFOAM as a third-party library to couple OpenFOAM with other solvers or OpenFOAM with itself.

Target audience

Users who want to couple OpenFOAM with other solvers or OpenFOAM with itself.

Proposal

A working general integration of MUI is proposed by applying the following patch to the OpenFOAM repository.

muiIntegrationOF.patch

Below is a summarise of what have been added and modified.

  • Configure MUI inclusion in OpenFOAM.

    • Added MUI related commands in the following files

      • bin/tools/foamConfigurePaths
      • etc/config.csh/functions
      • etc/config.csh/setup
      • etc/config.csh/unset
      • etc/config.sh/functions
      • etc/config.sh/setup
      • etc/config.sh/unset

    • Created the following files to setup MUI include through ThirdParty installation. These files also act as switches for MUI inclusion. MUI is enabled by setting mui_version=MUI-2.0 in these files and a global variable at compile time export FOAM_USE_MUI=1 will be set. MUI is disabled by setting mui_version=MUI-none in these files. MUI will be disabled by default.

      • etc/config.csh/mui
      • etc/config.sh/mui

  • There is a new folder named mpi-MUI in src/Pstream contains the MUI integrated Pstream. Make files src/Pstream/Allwmake and src/Pstream/Allwclean have been modified so that the newly created make files src/Pstream/Allwmake-mpi-MUI and src/Pstream/Allwclean-mpi-MUI can be called and mpi-MUI can be built once FOAM_USE_MUI=1 is set.

  • A new variable extern MPI_Comm commWorld_; has been added in src/Pstream/mpi/PstreamGlobals.H, src/Pstream/mpi/PstreamGlobals.C and src/Pstream/mpi/UPstream.C so that to facilitate the replacing of the default MPI_COMM_WORLD into the MUI enabled MPI communicator in mpi-MUI folder.

  • The code piece of MPI_Init_thread() in src/Pstream/mpi/UPstream.C is proposed to be separated into a header file mpiInitThread.H so that to facilitate the MUI library to setup the MPI communicator in mpi-MUI folder. In this way, there is no need to modify codes in src/Pstream/mpi/UPstream.C directly.

  • In mpi-MUI folder, there is only two header files src/Pstream/mpi-MUI/mpiInitThread.H that contains the MUI returned MPI communicator and src/Pstream/mpi-MUI/PstreamGlobals.H aiming to include mui.h header file. Once mpi-MUI folder is building, the src/Pstream/Allwclean-mpi-MUI will copy UPstream.C from mpi folder to mpi-MUI so that the two header files in mpi-MUI folder can be included. Other Pstream related header/source files will be used directly from the mpi folder as shown in src/Pstream/mpi-MUI/Make/files. The inclusion of -I../mpi has been put in src/Pstream/mpi-MUI/Make/options to enable the directly reuse of files in mpi folder. An alternative way to consider is to create soft links of these header/source files from the mpi folder.

  • Created a new general purpose header file in creating MUI coupling interfaces

    • src/OpenFOAM/include/createCouplingMUI.H

  • There is a new solver in applications/solvers/basic/laplacianFoamMUI that shows how to utilise the integration and the way to compile a solver is also shown in the corresponding Make folder options file. Key to this is the inclusion of a new mpi-MUI-rules which is located in wmake/rules/General and effectively replaces the usual mpi-rules normally used. It is mostly the same file but just adds an extra line at the bottom PINC += -I$(WM_THIRD_PARTY_DIR)/sources/mui/MUI-2.0/src. There is a new file wmake/scripts/have_mui to detection/setup of MUI.

  • There is a new tutorial case in tutorials/basic/laplacianFoamMUI that shows how to use the laplacianFoamMUI solver and a demo of couplingDict to set MUI coupling related variables.

  • There is a new test code in applications/test/coupling-MUI that provide a unit test on MUI integration.

  • Added MUI library related documentations in the following files in OpenFOAM

    • doc/Requirements.md

Related issue

ThirdParty issue #67

What does success look like, and how can we measure that?

The Patch has been tested with the OpenFOAM development repository (commit e651d635).

The proposed changes can be patched and tested as follows

  • Clone Development Repositories
git clone https://develop.openfoam.com/Development/ThirdParty-common.git
git clone https://develop.openfoam.com/Development/openfoam.git
  • Obtain the MUI source file in the ThirdParty Repository
cd ThirdParty-common/sources
mkdir mui && cd mui
wget https://github.com/MxUI/MUI/archive/refs/tags/2.0.tar.gz
tar -xf 2.0.tar.gz && rm 2.0.tar.gz

  • Obtain and place patches in Repositories

  • Patch

cd openfoam/
patch -p2 < muiIntegrationOF.patch
rm muiIntegrationOF.patch
cd ../ThirdParty-common/
patch -p2 < muiIntegrationTP.patch
rm muiIntegrationTP.patch

  • Change permission of newly added files if needed

  • Enable MUI support (MUI is disabled by default)

    • Modify L37 of openfoam/etc/config.sh to change mui_version=MUI-none into mui_version=MUI-2.0

  • Source and Allwmake

cd openfoam/
source etc/bashrc
./Allwmake -j 4
  • Test MUI enabled OpenFOAM
cd openfoam/applications/test/coupling-MUI
./testCase/Allrun
cd  openfoam/tutorials/basic/laplacianFoamMUI
./AllrunCoupled

If MUI library successfully integrated, the following log messages can be found for the coupling-MUI unit test.

...

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

If MUI library successfully integrated, the following log messages can be found for the laplacianFoamMUI tutorial.

....

MUI [lib_mpi_multidomain]: Rank: 2, "domain2" registered interface "interface_twoD_1" as 59a4e385
MUI [lib_mpi_multidomain]: Rank: 3, "domain2" registered interface "interface_twoD_1" as 59a4e385
MUI Info [lib_mpi_multidomain]: 1 distinct interface(s) found
MUI [lib_mpi_multidomain]: Setting up interface interface_twoD_1 [59a4e385] (rank ids are local to each interface)
MUI [comm_mpi.h]: Rank: 3, Identifier: mpi://domain2/interface_twoD_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 2, Identifier: mpi://domain2/interface_twoD_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 1, Identifier: mpi://domain1/interface_twoD_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 0, Identifier: mpi://domain1/interface_twoD_1, Domain size: 2, Peers: 2
MUI [lib_mpi_multidomain]: Rank: 2, "domain2" registered interface "interface_threeD_1" as 31f80b7e
MUI [lib_mpi_multidomain]: Rank: 1, "domain1" registered interface "interface_threeD_1" as 31f80b7e
MUI [lib_mpi_multidomain]: Rank: 3, "domain2" registered interface "interface_threeD_1" as 31f80b7e
MUI [lib_mpi_multidomain]: Rank: 0, "domain1" registered interface "interface_threeD_1" as 31f80b7e
MUI Info [lib_mpi_multidomain]: 1 distinct interface(s) found
MUI [lib_mpi_multidomain]: Setting up interface interface_threeD_1 [31f80b7e] (rank ids are local to each interface)
MUI [comm_mpi.h]: Rank: 0, Identifier: mpi://domain1/interface_threeD_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 3, Identifier: mpi://domain2/interface_threeD_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 2, Identifier: mpi://domain2/interface_threeD_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 1, Identifier: mpi://domain1/interface_threeD_1, Domain size: 2, Peers: 2
MUI [lib_mpi_multidomain]: Rank: 2, "domain2" registered interface "interface_T_1" as 4a5523ab
MUI [lib_mpi_multidomain]: Rank: 1, "domain1" registered interface "interface_T_1" as 4a5523ab
MUI [lib_mpi_multidomain]: Rank: 0, "domain1" registered interface "interface_T_1" as 4a5523ab
MUI [lib_mpi_multidomain]: Rank: 3, "domain2" registered interface "interface_T_1" as 4a5523ab
MUI Info [lib_mpi_multidomain]: 1 distinct interface(s) found
MUI [lib_mpi_multidomain]: Setting up interface interface_T_1 [4a5523ab] (rank ids are local to each interface)
MUI [comm_mpi.h]: Rank: 0, Identifier: mpi://domain1/interface_T_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 1, Identifier: mpi://domain1/interface_T_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 3, Identifier: mpi://domain2/interface_T_1, Domain size: 2, Peers: 2
MUI [comm_mpi.h]: Rank: 2, Identifier: mpi://domain2/interface_T_1, Domain size: 2, Peers: 2

Calculating temperature distribution


Calculating temperature distribution

Time = 0.005

Time = 0.005

MUI interface "domain1"/"interface_twoD_1" value committed: 1 at Iteration = 0
MUI interface "domain2"/"interface_twoD_1" value committed: 1 at Iteration = 0
MUI interface "domain1"/"interface_threeD_1" value committed: 2 at Iteration = 0
MUI interface "domain1"/"interface_T_1" value committed: 3 at Iteration = 0
MUI interface "domain2"/"interface_threeD_1" value committed: 2 at Iteration = 0
MUI interface "domain2"/"interface_T_1" value committed: 3 at Iteration = 0
DICPCG:  Solving for T, Initial residual = 1, Final residual = 8.33243e-07, No Iterations 7
DICPCG:  Solving for T, Initial residual = 0.00446911, Final residual = 7.14892e-07, No Iterations 4
DICPCG:  Solving for T, Initial residual = 0.000148123, Final residual = 6.63323e-07, No Iterations 2
MUI interface "domain1"/"interface_twoD_1" value fetched: 1 at Iteration = 0
MUI interface "domain1"/"interface_threeD_1" value fetched: 2 at Iteration = 0
MUI interface "domain1"/"interface_T_1" value fetched: 3 at Iteration = 0
ExecutionTime = 0.07 s  ClockTime = 0 s

Time = 0.01

MUI interface "domain1"/"interface_twoD_1" value committed: 1 at Iteration = 1
MUI interface "domain1"/"interface_threeD_1" value committed: 2 at Iteration = 1
MUI interface "domain1"/"interface_T_1" value committed: 3 at Iteration = 1
DICPCG:  Solving for T, Initial residual = 0.203755, Final residual = 1.57027e-07, No Iterations 7
DICPCG:  Solving for T, Initial residual = 0.00184337, Final residual = 2.32416e-07, No Iterations 4
DICPCG:  Solving for T, Initial residual = 1, Final residual = 8.33243e-07, No Iterations 7
DICPCG:  Solving for T, Initial residual = 5.70921e-05, Final residual = 2.8776e-07, No Iterations 2
DICPCG:  Solving for T, Initial residual = 0.00446911, Final residual = 7.14892e-07, No Iterations 4
DICPCG:  Solving for T, Initial residual = 0.000148123, Final residual = 6.63323e-07, No Iterations 2
MUI interface "domain2"/"interface_twoD_1" value fetched: 1 at Iteration = 0
MUI interface "domain2"/"interface_threeD_1" value fetched: 2 at Iteration = 0
MUI interface "domain2"/"interface_T_1" value fetched: 3 at Iteration = 0
ExecutionTime = 0.1 s  ClockTime = 0 s

Time = 0.01

MUI interface "domain2"/"interface_twoD_1" value committed: 1 at Iteration = 1
MUI interface "domain2"/"interface_threeD_1" value committed: 2 at Iteration = 1
MUI interface "domain2"/"interface_T_1" value committed: 3 at Iteration = 1
MUI interface "domain1"/"interface_twoD_1" value fetched: 1 at Iteration = 1
MUI interface "domain1"/"interface_threeD_1" value fetched: 2 at Iteration = 1
MUI interface "domain1"/"interface_T_1" value fetched: 3 at Iteration = 1
ExecutionTime = 0.08 s  ClockTime = 0 s

Time = 0.015

MUI interface "domain1"/"interface_twoD_1" value committed: 1 at Iteration = 2
MUI interface "domain1"/"interface_threeD_1" value committed: 2 at Iteration = 2
MUI interface "domain1"/"interface_T_1" value committed: 3 at Iteration = 2
DICPCG:  Solving for T, Initial residual = 0.109922, Final residual = 4.92455e-07, No Iterations 6
DICPCG:  Solving for T, Initial residual = 0.00104616, Final residual = 7.0666e-07, No Iterations 3
DICPCG:  Solving for T, Initial residual = 3.13839e-05, Final residual = 1.66653e-07, No Iterations 2
DICPCG:  Solving for T, Initial residual = 0.203755, Final residual = 1.57027e-07, No Iterations 7
DICPCG:  Solving for T, Initial residual = 0.00184337, Final residual = 2.32416e-07, No Iterations 4
DICPCG:  Solving for T, Initial residual = 5.70921e-05, Final residual = 2.8776e-07, No Iterations 2
MUI interface "domain2"/"interface_twoD_1" value fetched: 1 at Iteration = 1
MUI interface "domain2"/"interface_threeD_1" value fetched: 2 at Iteration = 1
MUI interface "domain2"/"interface_T_1" value fetched: 3 at Iteration = 1
ExecutionTime = 0.11 s  ClockTime = 1 s

Time = 0.015

MUI interface "domain2"/"interface_twoD_1" value committed: 1 at Iteration = 2
MUI interface "domain2"/"interface_threeD_1" value committed: 2 at Iteration = 2
MUI interface "domain2"/"interface_T_1" value committed: 3 at Iteration = 2
DICPCG:  Solving for T, Initial residual = 0.109922, Final residual = 4.92455e-07, No Iterations 6
MUI interface "domain1"/"interface_twoD_1" value fetched: 1 at Iteration = 2
MUI interface "domain1"/"interface_threeD_1" value fetched: 2 at Iteration = 2
MUI interface "domain1"/"interface_T_1" value fetched: 3 at Iteration = 2
ExecutionTime = 0.1 s  ClockTime = 1 s

...
Edited by Wendi Liu