README.md

@@ -7,46 +7,119 @@ and its external dependencies (i.e. [Hypre][Hypre]) into arbitrary OpenFOAM simu
 
 The library provides:
 - ldu2csr matrix conversion.
-- selection of solvers and preconditioners available in [Petsc] 
+- selection of solvers and preconditioners available in [Petsc]
   and in its external libraries (i.e. [Hypre]).
 - editing of run-time options by a dictionary and/or a rc file.
 - basic caching implementation of a Petsc matrix.
 
 
+## License
+
+The source code license: GPL-3.0-or-later
+
+
 ## Requirements
 
-1. [OpenFOAM-v1906] or higher or a recent [development version][OpenFOAM-git]
+1. [OpenFOAM-v1912] or newer, or a recent [development version][OpenFOAM-git]
    from [OpenFOAM.com][OpenFOAM].
 2. [Petsc] 3.10 or newer,
    optionally compiled with [Hypre] support (optionally with MPI support).
 
+
 ## Building
 
-Ensure that the OpenFOAM environment is active and that Petsc 
+Ensure that the OpenFOAM environment is active and that Petsc
 can be found (check that the `PETSC_ARCH_PATH` environment variable is properly set).
 Read [How to build Petsc for OpenFOAM][Petsc-Installation] for more information
-or visit the Petsc website [How to build Petsc][Petsc-Ext-Installation]. 
+or visit the Petsc website [How to build Petsc][Petsc-Ext-Installation].
+
 
 ### To install in the normal OpenFOAM directories (using `wmake`)
 
-Simply use the supplied `Allwmake` script:
-````
+Using the supplied `Allwmake` script without arguments:
+```
 ./Allwmake
-````
-This will install the library under
-`FOAM_USER_LIBBIN`.
-
-
-## Authors
+```
+will install the library under `FOAM_MODULE_LIBBIN`.
+
+If such variable is not defined, the standard `FOAM_USER_LIBBIN` will be used.
+
+To install into different locations, you can use the `-prefix=PATH` or
+`-prefix=shortcut` option (for OpenFOAM-v2006 and newer):
+```
+./Allwmake -prefix=/install/path
+
+# Installing into the OpenFOAM FOAM_LIBBIN
+./Allwmake -prefix=openfoam
+```
+or specify via the environment.
+```
+FOAM_MODULE_LIBBIN=/install/path  ./Allwmake
+```
+
+## How to use it
+
+In order to use the library, two changes are required:
+- add the libpetscFoam library to the optional keyword entry libs of the control dict file
+
+      libs       (petscFoam);
+
+- set the name of the solver and preconditioner in each solver of the fvSolution to petsc.
+  The options database keys of each PETSc object have to be added in the petsc/options subdict of each solver equation.
+  The default behaviour of the library is to convert the matrix from LDU to CSR at each time step.
+  However, the user can change the cache update frequency among the following choices:
+
+  - never (none)
+  - always
+  - periodic
+  - adaptive
+
+  The cache update frequency is set for both matrix and preconditioner in the petsc/caching subdict.
+
+An example is reported below. Other examples can be found in the tutorial folder or in the [HPC repo](https://develop.openfoam.com/committees/hpc).
+For more details, the user can read the paper [1].
+
+    solvers
+    {
+        p
+        {
+            solver          petsc;
+            preconditioner  petsc;
+
+            petsc
+            {
+                options
+                {
+                    ksp_type    cg;
+                    pc_type     bjacobi;
+                    sub_pc_type icc;
+                }
+
+                caching
+                {
+                    matrix
+                    {
+                        update always;
+                    }
+
+                    preconditioner
+                    {
+                        update always;
+                    }
+                }
+            }
+        }
+    }
+
+
+## Authors / Contributors
+
+| Name | Affiliation | Email
+|------|-------|-----------|
+| Mark Olesen  | ESI-OpenCFD | |
+| Simone Bna   | CINECA | simone.bna@cineca.it |
+| Stefano Zampini | KAUST | |
 
-- Mark Olesen | <mark.olesen@esi-group.com>  | (ESI-OpenCFD)
-- Simone Bna  | <simone.bna@cineca.it>       | (CINECA)
-
-
-## License:
-
-Licensed under GNU General Public License <http://www.gnu.org/licenses/>
-with the same terms as OpenFOAM itself.
 
 ----
 
@@ -58,3 +131,9 @@ with the same terms as OpenFOAM itself.
 [Petsc]: https://www.mcs.anl.gov/petsc/
 [Petsc-Installation]: doc/README.md
 [Petsc-Ext-Installation]: https://www.mcs.anl.gov/petsc/documentation/installation.html
+
+
+## References
+
+ [1] S. Bnà, I. Spisso, M. Olesen, G. Rossi *PETSc4FOAM: A Library to plug-in PETSc into the OpenFOAM
+Framework* [PRACE White paper](https://prace-ri.eu/wp-content/uploads/WP294-PETSc4FOAM-A-Library-to-plug-in-PETSc-into-the-OpenFOAM-Framework.pdf)

doc/README.md

@@ -4,7 +4,8 @@ Configure and install PETSc.
 For example,
 ```
 ./configure
-    --with-64-bit-indices=0 --with-precision=double \
+    --with-64-bit-indices=0 \
+    --with-precision=double \
     --prefix=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER$WM_PRECISION_OPTION$WM_LABEL_OPTION/petsc-git \
     PETSC_ARCH=$WM_OPTIONS
 
@@ -16,14 +17,18 @@ Note that this approach does not account for mixing of mpi
 distributions or versions.
 
 Export the appropriate PETSC_ARCH_PATH location and/or adjust config
-files(s): `$WM_PROJECT_DIR/etc/config.sh/petsc` or
-`$HOME/.OpenFOAM/config.sh/petsc`.
+files(s):
+
+- global: `$WM_PROJECT_DIR/etc/config.sh/petsc`
+- user:   `$HOME/.OpenFOAM/config.sh/petsc`
+
 
 
 Another possible configuration
 ```
 ./configure
-    --with-64-bit-indices=0 --with-precision=double \
+    --with-64-bit-indices=0 \
+    --with-precision=double \
     --prefix=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER$WM_PRECISION_OPTION$WM_LABEL_OPTION/petsc-git \
     PETSC_ARCH=$WM_OPTIONS \
     --download-f2cblaslapack \
@@ -31,6 +36,8 @@ Another possible configuration
     --with-mpi-dir=$MPI_ARCH_PATH
 ```
 
+The ThirdParty [makePETSC][makePETSC] script may be of some use.
+
 # Running
 
 Requires LD_LIBRARY_PATH to include PETSc information.
@@ -38,3 +45,19 @@ For example, using the config settings:
 ```
 eval $(foamEtcFile -sh -config petsc -- -force)
 ```
+
+And an additional library loading in the `system/controlDict`.
+For example,
+```
+libs    (petscFoam);
+```
+Alternatively, can preload this library directly from the command-lib
+for the solver.
+For example,
+```
+simpleFoam -lib petscFoam ...
+```
+
+----
+
+[makePETSC]: https://develop.openfoam.com/Development/ThirdParty-common/-/blob/master/makePETSC

src/Allwmake

 #!/bin/sh
 cd "${0%/*}" || exit                            # Run from this directory
+. "${WM_PROJECT_DIR:?}"/wmake/scripts/AllwmakeParseArguments
 #------------------------------------------------------------------------------
 
 petsc4Foam/Allwmake $*

src/petsc4Foam/Allwclean

@@ -7,7 +7,7 @@ rm -f $FOAM_LIBBIN/libpetscFoam*                 # Cleanup library
 rm -f $FOAM_SITE_LIBBIN/libpetscFoam*            # ... extra safety
 rm -f $FOAM_USER_LIBBIN/libpetscFoam*            # ... extra safety
 
-# Cleanup generated files - remove entire top-level 
+# Cleanup generated files - remove entire top-level
 removeObjectDir "$PWD"
 
 # -----------------------------------------------------------------------------

src/petsc4Foam/Allwmake

@@ -11,16 +11,19 @@ then
     echo "wmake(petsc) $targetType : $PETSC_ARCH_PATH"
     wmake $targetType
 
-    /bin/cat<<INFO 1>&2
+    /bin/cat<<INFORMATION 1>&2
 ==> Before running, verify that PETSc libraries can be found
+For example,
 
-Enable in the OpenFOAM etc/bashrc, define manually or try with the
-following (POSIX shell):
+    foamHasLibrary -verbose petscFoam
+
+Define manually, enable in OpenFOAM etc/bashrc, or try the following [POSIX]:
 
     eval \$(foamEtcFile -sh -config petsc -- -force)
 
 ==
-INFO
+INFORMATION
+
 fi
 
 #------------------------------------------------------------------------------

src/petsc4Foam/Make/options

@@ -12,11 +12,10 @@ EXE_INC = \
     $(PFLAGS) $(PINC) \
     -Wno-old-style-cast \
     -I$(LIB_SRC)/finiteVolume/lnInclude \
-    -I$(LIB_SRC)/fvOptions/lnInclude \
     -I$(LIB_SRC)/meshTools/lnInclude \
-    -I${PETSC_INC_DIR}
+    $(foreach dir,$(PETSC_INC_DIR),-I$(dir))
 
 LIB_LIBS = \
     -lfiniteVolume \
-    -L$(FOAM_EXT_LIBBIN)/$(FOAM_MPI) \
-    -L${PETSC_LIB_DIR} -lpetsc
+    -lmeshTools \
+    $(foreach dir,$(PETSC_LIB_DIR),-L$(dir)) -lpetsc

src/petsc4Foam/utils/petscCacheManager.C

@@ -32,15 +32,15 @@ License
 
 const Foam::Enum
 <
-    Foam::PetscUtils::Caching::cachingTypes
+    Foam::PetscUtils::Caching::updateTypes
 >
-Foam::PetscUtils::Caching::cachingTypeNames_
+Foam::PetscUtils::Caching::updateTypeNames_
 {
-    { Caching::None, "none" },
-    { Caching::None, "never" }, // Alias
-    { Caching::Always, "always" },
-    { Caching::Periodic, "periodic" },
-    { Caching::Adaptive, "adaptive" },
+    { updateTypes::Always, "always" },
+    { updateTypes::Periodic, "periodic" },
+    { updateTypes::Adaptive, "adaptive" },
+    { updateTypes::Never, "never" },
+    { updateTypes::Never, "none" }, // Alias
 };
 
 

src/petsc4Foam/utils/petscCacheManager.H

@@ -40,7 +40,7 @@ Description
         {
             matrix
             {
-                update  always;
+                update  always;  // (default: always)
             }
 
             preconditioner
@@ -83,24 +83,26 @@ struct Caching
 {
     // Public Data Types
 
-        //- Supported caching types
-        enum cachingTypes
+        //- Caching update types
+        enum updateTypes
         {
-            None,
-            Always,
-            Periodic,
-            Adaptive
+            Always,         //!< "always" update every time-step [default]
+            Periodic,       //!< "periodic" update at given period
+            Adaptive,       //!< "adaptive" update scheme
+            Never,          //!< "never" update (or "none")
         };
 
 
-        //- Names for selectable caching types
-        static const Enum<cachingTypes> cachingTypeNames_;
+        //- Names for the update types
+        static const Enum<updateTypes> updateTypeNames_;
 
 
     // Member Data
 
-        cachingTypes updateType_;
+        //- Caching update
+        updateTypes updateType_;
 
+        //- Cache update frequency (for Periodic)
         int updateFreq_;
 
         //- Elapsed time (s) for current iteration
@@ -118,10 +120,10 @@ struct Caching
 
     // Constructors
 
-        //- Default construct. No caching
+        //- Default construct. Always update (no caching).
         Caching()
         :
-           updateType_{cachingTypes::None},
+           updateType_{updateTypes::Always},
            updateFreq_{1},
            timeIter{0},
            time0{0},
@@ -133,17 +135,18 @@ struct Caching
     {
         dictionary cacheDict = dict.subOrEmptyDict(key);
         updateType_ =
-            cachingTypeNames_.getOrDefault
+            updateTypeNames_.getOrDefault
             (
                 "update",
                 cacheDict,
-                cachingTypes::None
+                updateTypes::Always,
+                false  // non-failsafe - Fatal on bad enumeration
             );
 
-        if (updateType_ == cachingTypes::Periodic)
+        if (updateType_ == updateTypes::Periodic)
         {
-            dictionary coeffsDict = cacheDict.subOrEmptyDict("periodicCoeffs");
-            updateFreq_ = coeffsDict.getOrDefault<int>("frequency", 1);
+            dictionary coeffs(cacheDict.subOrEmptyDict("periodicCoeffs"));
+            updateFreq_ = coeffs.getOrDefault<int>("frequency", 1);
         }
     }
 };
@@ -243,7 +246,7 @@ private:
     {
         switch (caching.updateType_)
         {
-            case PetscUtils::Caching::None:
+            case PetscUtils::Caching::Never:
             {
                 return false;
                 break;
@@ -279,7 +282,7 @@ private:
                     const double ratio1 =
                         precondCaching.time1 / precondCaching.timeIter;
 
-                    int nsteps =
+                    const int nsteps =
                         min(1e5, ratio0 * (1. / mag(1. - ratio1 + 1e-6)));
 
                     if (iter >= nsteps)
@@ -293,7 +296,7 @@ private:
             }
         }
 
-        // Default
+        // Default: Never
         return false;
     }
 };

tutorials/Allclean

+#!/bin/sh
+cd "${0%/*}" || exit                            # Run from this directory
+. ${WM_PROJECT_DIR:?}/bin/tools/CleanFunctions  # Tutorial clean functions
+. ${WM_PROJECT_DIR:?}/bin/tools/LogFunctions    # Tutorial logfile functions
+#------------------------------------------------------------------------------
+
+echo "--------"
+removeLogs
+
+echo "Cleaning tutorials"
+foamCleanTutorials -self
+
+echo "--------"
+
+#------------------------------------------------------------------------------

tutorials/Allcollect

+#!/bin/sh
+cd "${0%/*}" || exit                            # Run from this directory
+. ${WM_PROJECT_DIR:?}/bin/tools/LogFunctions    # Tutorial log-file functions
+#------------------------------------------------------------------------------
+
+# Collect log files as 'testLoopReport'
+collectLogs
+
+#------------------------------------------------------------------------------

tutorials/Allrun

+#!/bin/sh
+cd "${0%/*}" || exit                            # Run from this directory
+## . ${WM_PROJECT_DIR:?}/bin/tools/LogFunctions    # Tutorial logfile functions
+#------------------------------------------------------------------------------
+
+# Test for petscFoam
+if command -v foamHasLibrary >/dev/null \
+&& ! foamHasLibrary -verbose petscFoam
+then
+    /bin/cat<<INFORMATION 1>&2
+
+==> petsc not found?
+Define manually, enable in OpenFOAM etc/bashrc, or try the following [POSIX]:
+
+    eval \$(foamEtcFile -sh -config petsc -- -force)
+
+==
+INFORMATION
+
+    echo "Skip petscFoam tutorials"
+    exit 1
+fi
+
+foamRunTutorials -skipFirst $*          # Run tutorials recursively
+
+## collectLogs
+
+#------------------------------------------------------------------------------

tutorials/basic/laplacianFoam/flange/0/T

-/*--------------------------------*- C++ -*----------------------------------*\
-| =========                 |                                                 |
-| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
-|  \\    /   O peration     | Version:  v1806                                 |
-|   \\  /    A nd           | Web:      www.OpenFOAM.com                      |
-|    \\/     M anipulation  |                                                 |
-\*---------------------------------------------------------------------------*/
-FoamFile
-{
-    version     2.0;
-    format      ascii;
-    class       volScalarField;
-    object      T;
-}
-// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
-
-dimensions      [0 0 0 1 0 0 0];
-
-internalField   uniform 273;
-
-boundaryField
-{
-    patch1
-    {
-        type            zeroGradient;
-    }
-
-    patch2
-    {
-        type            fixedValue;
-        value           uniform 273;
-    }
-
-    patch3
-    {
-        type            zeroGradient;
-    }
-
-    patch4
-    {
-        type            fixedValue;
-        value           uniform 573;
-    }
-}
-
-// ************************************************************************* //

tutorials/basic/laplacianFoam/flange/Allclean

 #!/bin/sh
-cd "${0%/*}" || exit                                # Run from this directory
-. ${WM_PROJECT_DIR:?}/bin/tools/CleanFunctions      # Tutorial clean functions
+cd "${0%/*}" || exit                            # Run from this directory
+. ${WM_PROJECT_DIR:?}/bin/tools/CleanFunctions  # Tutorial clean functions
 #------------------------------------------------------------------------------
 
-cleanCase
+rm -rf run
+
+cleanCase0
 
 #------------------------------------------------------------------------------

tutorials/basic/laplacianFoam/flange/Allrun

 #!/bin/sh
 cd "${0%/*}" || exit                                # Run from this directory
-. ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions        # Tutorial run functions
 #------------------------------------------------------------------------------
 
-runApplication ansysToFoam flange.ans -scale 0.001
-runApplication $(getApplication)
-# runApplication foamToEnsight -noZero
-# runApplication foamToVTK
+./Allrun.init
+
+dst=run
+
+if [ -d "$dst" ]
+then
+    (cd "$dst" && foamRunTutorials)
+
+    # Copy back log files
+    cp -f "$dst"/log.* .
+fi
 
 #------------------------------------------------------------------------------

tutorials/basic/laplacianFoam/flange/Allrun-parallel

 #!/bin/sh
 cd "${0%/*}" || exit                                # Run from this directory
-. ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions        # Tutorial run functions
 #------------------------------------------------------------------------------
 
-runApplication ansysToFoam flange.ans -scale 0.001
-runApplication decomposePar
-runParallel $(getApplication)
+./Allrun.init
+
+dst=run
+
+if [ -f "$dst/Allrun-parallel" ]
+then
+    "$dst"/Allrun-parallel
+
+    # Copy back log files
+    cp -f "$dst"/log.* .
+fi
 
 #------------------------------------------------------------------------------

tutorials/basic/laplacianFoam/flange/Allrun.init

+#!/bin/sh
+cd "${0%/*}" || exit                            # Run from this directory
+. ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions    # Tutorial run functions
+#------------------------------------------------------------------------------
+tutorial="basic/laplacianFoam/flange"
+
+src="$FOAM_TUTORIALS/$tutorial"
+dst=run
+
+if [ -d "$dst" ]
+then
+
+    echo "Already initialized $tutorial -> $dst/"
+
+elif [ -d "$src" ]
+then
+
+    echo "Copying $tutorial -> $dst/"
+
+    mkdir "$dst"
+    cp -r "$src"/constant "$dst"
+    cp -r "$src"/system "$dst"
+
+    # Copy zero directory (0.orig/, or 0/)
+    for i in 0.orig 0
+    do
+        if [ -d "$src/$i" ]
+        then
+            cp -r "$src/$i" "$dst"
+            break
+        fi
+    done
+
+    # Allclean, Allrun etc may not exist
+    for i in "$src"/All*
+    do
+        if [ -f "$i" ]
+        then
+            cp "$i" "$dst"
+        fi
+    done
+
+else
+
+    echo "No OpenFOAM tutorial: $tutorial"
+    exit 2
+fi
+
+
+#------------------------------------------------------------------------------
+
+# Adjust controlDict - needs refinement if controlDict has other libs
+if [ -f "$dst"/system/controlDict ] \
+&& ! grep -q petscFoam "$dst"/system/controlDict
+then
+    echo "Add libs (petscFoam) to controlDict"
+    echo "libs (petscFoam);" >> "$dst"/system/controlDict
+fi
+
+# Use fvSolution-petsc
+if [ -f "$dst"/system/fvSolution ] \
+&& [ ! -f "$dst"/system/fvSolution-petsc ]
+then
+    echo "Rename fvSolution and relink to fvSolution-petsc"
+    mv "$dst"/system/fvSolution "$dst"/system/fvSolution-foam
+    (cd "$dst"/system && ln -sf ../../fvSolution-petsc fvSolution)
+fi
+
+#------------------------------------------------------------------------------

tutorials/basic/laplacianFoam/flange/constant/fvOptions-bak

-/*--------------------------------*- C++ -*----------------------------------*\
-| =========                 |                                                 |
-| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
-|  \\    /   O peration     | Version:  v1806                                 |
-|   \\  /    A nd           | Web:      www.OpenFOAM.com                      |
-|    \\/     M anipulation  |                                                 |
-\*---------------------------------------------------------------------------*/
-FoamFile
-{
-    version     2.0;
-    format      ascii;
-    class       dictionary;
-    location    "constant";
-    object      fvOptions;
-}
-// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
-
-view
-{
-    type    petscViewMatrix;
-    libs    ("libpetscFoam.so");
-
-    active          on;
-
-    timeStart       0.1;
-    duration        10;
-    selectionMode   all;
-
-    fields          (T);
-    sleep           3;
-}
-
-// ************************************************************************* //

tutorials/basic/laplacianFoam/flange/constant/transportProperties

-/*--------------------------------*- C++ -*----------------------------------*\
-| =========                 |                                                 |
-| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
-|  \\    /   O peration     | Version:  v1806                                 |
-|   \\  /    A nd           | Web:      www.OpenFOAM.com                      |
-|    \\/     M anipulation  |                                                 |
-\*---------------------------------------------------------------------------*/
-FoamFile
-{
-    version     2.0;
-    format      ascii;
-    class       dictionary;
-    location    "constant";
-    object      transportProperties;
-}
-// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
-
-DT              4e-05;
-
-
-// ************************************************************************* //

tutorials/basic/laplacianFoam/flange/system/fvSolution-petsc → tutorials/basic/laplacianFoam/flange/fvSolution-petsc

 /*--------------------------------*- C++ -*----------------------------------*\
 | =========                 |                                                 |
 | \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
-|  \\    /   O peration     | Version:  v1806                                 |
-|   \\  /    A nd           | Web:      www.OpenFOAM.com                      |
+|  \\    /   O peration     | Version:  v2006                                 |
+|   \\  /    A nd           | Website:  www.openfoam.com                      |
 |    \\/     M anipulation  |                                                 |
 \*---------------------------------------------------------------------------*/
 FoamFile
@@ -37,12 +37,26 @@ solvers
                 // ksp_norm_type natural;
                 mat_type  mpiaij;
             }
+
             // use petsc default convergence testing instead of OpenFOAM's L1 based criterion
             // default is false
             use_petsc_residual_norm false;
             // monitor (print to stdout) residual reduction in OpenFOAM norm
             // default is false
             monitor_foam_residual_norm false;
+
+            caching
+            {
+                matrix
+                {
+                    update always;
+                }
+
+                preconditioner
+                {
+                    update always;
+                }
+            }
         }
 
         tolerance       1e-06;