fixedJumpAMI (cyclicAMI) no longer runs in parallel

The fixedJumpAMI boundary condition can no longer run in parallel.

Steps to reproduce

Create a case using this boundary condition and run it in parallel.

Example case

An example case is added to this bug report. This case consists in a 3-periodic cube using fixedJumpAMI along the x direction. Other directions are cyclicAMI. The solver is simpleFoam.

What is the current bug behaviour?

Computation crashes on a FATAL ERROR.

What is the expected correct behavior?

The computation should run in parallel similar to single core (no crash).

Relevant logs and/or images

simpleFoam crashes in parallel with this kind of message (eg. for processor [1]) :

[1]
[1]
[1] --> FOAM FATAL ERROR: (openfoam-2412)
**[1] Outstanding recv request(s) on** patch xMax field p
[1] 
[1]     From void Foam::cyclicAMIFvPatchField<Type>::initInterfaceMatrixUpdate(Foam::solveScalarField&, bool, const Foam::lduAddressing&, Foam::label, const solveScalarField&, const scalarField&, Foam::direction, Foam::UPstream::commsTypes) const [with Type = double; Foam::solveScalarField = Foam::Field<double>; Foam::label = int; Foam::scalarField = Foam::Field<double>; Foam::direction = unsigned char]
[1]     in file fields/fvPatchFields/constraint/cyclicAMI/cyclicAMIFvPatchField.C at line 600.
[1] 
[1] 
[1] [stack trace]
[1] =============
[1] #5  [0] #5  Foam::lduMatrix::Amul(Foam::Field<double>&, Foam::tmp<Foam::Field<double> > const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char) constFoam::lduMatrix::Amul(Foam::Field<double>&, Foam::tmp<Foam::Field<double> > const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char) const[2] #5  Foam::lduMatrix::Amul(Foam::Field<double>&, Foam::tmp<Foam::Field<double> > const&, Foam::FieldField<Foam::Field, double> const&, Foam::UPtrList<Foam::lduInterfaceField const> const&, unsigned char) const in ~/OpenFOAM/OpenFOAM-plus/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[1] #8  Foam::GAMGSolver::solve(Foam::Field<double>&, Foam::Field<double> const&, unsigned char) const in ~/OpenFOAM/OpenFOAM-plus/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[1] #9  Foam::fvMatrix<double>::solveSegregated(Foam::dictionary const&) in  in ~/OpenFOAM/OpenFOAM-plus/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so
[1] #10  Foam::fvMatrix<double>::solveSegregatedOrCoupled(Foam::dictionary const&)[2] #10  Foam::fvMatrix<double>::solveSegregatedOrCoupled(Foam::dictionary const&)[0] #10  Foam::fvMatrix<double>::solveSegregatedOrCoupled(Foam::dictionary const&) in ~/OpenFOAM/OpenFOAM-plus/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
[1] #11[0] #11    Foam::fvMesh::solve(Foam::fvMatrix<double>&, Foam::dictionary const&) constFoam::fvMesh::solve(Foam::fvMatrix<double>&, Foam::dictionary const&) const[2] #11  Foam::fvMesh::solve(Foam::fvMatrix<double>&, Foam::dictionary const&) const in ~/OpenFOAM/OpenFOAM-plus/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
[1] #12   in ~/OpenFOAM/OpenFOAM-plus/platforms/linux64GccDPInt32Opt/lib/libfiniteVolume.so
[1] #14   in /usr/lib64/libc.so.6
[1] =============

Environment information

• OpenFOAM version : v2412
• Operating system : RedHat (Rocky 8)
• Hardware info : GenuineIntel platform
• Compiler : gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-16)

Possible fixes

The source code that might be responsible to this crash is : src/finiteVolume/fields/fvPatchFields/constraint/cyclicAMI/cyclicAMIFvPatchField.C

In particular, commit 8b85e5c9 added assertions "that all receives are known to have finished". But the cyclicAMIFvPatchField.C crashes on these additions ...

cubeBugFixedJumpAMI.tar

If the uniformJumpAMI boundary condition is used, instead of fixedJumpAMI, then the error is the same. The case runs fine with v2206 and v2306.

changed the description

assigned to @Mattijs

The problem seems to be related to the jump classes derivated from cyclicAMI boundary conditions. Simulations will run without problems when using cyclicAMI but not when using the derived jump boundary conditions. Any hints on how to solve this issue are welcome to propose a solution to this bug issue ...

• develop works fine. v2412 does not.
• switching off localConsistency (in etc/controlDict) does not help.

changed the description

related to #3292 (closed) ?

Hi @Lionel.Gamet

Still puzzling why exactly the change 6520d8e0 would appear necessary (and why overlapping communication fails otherwise), but before spending too much time digging, could you check if a recent develop (that includes 6520d8e0) fixes or doesn't fix your problem and we can go again from there.

Commit 6520d8e0 was already present in the tested release.

That's not the answer we were hoping for!

Will give a test to "develop" branch, currently compiling... This should work, as Mattijs said. There are some changes on cyclicAMI and jumpAMI classes between "develop" and "master" branches.

The cube case runs fine with "develop" branch. The resulting flow field is however not symmetric nor exactly constant.

Setting localConsistency to 0 in controlDict did not change the behaviour :

OptimisationSwitches
{
    //- Enable enforced consistency of constraint bcs after 'local' operations.
    //  Default is on. Set to 0/false to revert to <v2306 behaviour
    localConsistency 0;
}

What happens if you add after

p.relax();

a

p.correctLocalBoundaryConditions();

? Does the flow field look better? Could be #3211 biting us as well. Hadn't realised the case was a simpleFoam one.

No change with p.correctLocalBoundaryConditions();

However, I have noticed that the velocity is rather constant over the domain, at each time step (with a min/max relative difference of the order 1e-7). But its value keeps increasing with time. This could appear abnormal but as there is no solid body in the system, there is no friction/dissipation and the average velocity keeps increasing.

However, the pressure gradient is linear and steady state.

Pressure boundary conditions : xMax { type fixedJumpAMI; patchType cyclicAMI; jump uniform 1; value $internalField; } xMin { type fixedJumpAMI; patchType cyclicAMI; jump uniform -1; value $internalField; }

Velocity boundary condition : xMin { type cyclicAMI; value $internalField; } xMax { type cyclicAMI; value $internalField; }

As a conclusion ... the solution is not stupid. The original point/bug was that this cannot run with v2412 (branch master) but can run with develop. This is very certainly due to changes in some of these files :

• src/finiteVolume/fields/fvPatchFields/constraint/cyclicAMI/cyclicAMIFvPatchField.C
• src/finiteVolume/fields/fvPatchFields/constraint/jumpCyclicAMI/jumpCyclicAMIFvPatchField.[CH]
• src/finiteVolume/fvMesh/fvPatches/constraint/cyclicAMI/cyclicAMIFvPatch.C

Which files would you advice to put back in master branch for testing ?

Problem solved when integrating these 2 files :

src/finiteVolume/fields/fvPatchFields/constraint/jumpCyclicAMI/jumpCyclicAMIFvPatchField.[CH]

from branch "develop" to "master".

Aha. The v2412 got the influence of the jump through the virtual patchNeighbourField(). (jumpCyclicAMIFvPatchField::patchNeighbourField() adds/subtracts the jump). With the new(ish) caching however there is no longer a call to the patchNeighbourField since it is done as a two-step process (starting send in initEvaluate, blocking & consuming in evaluate). With the develop (#3292 (closed)) it bypasses the cyclicAMI versions.

Not sure to fully understand your comment... Does it mean the new code prevents a bug that was present in the old code ? Does it reveal that the virtual patchNeighbourField() was faulty ?

• the virtual jumpCyclicAMI patchNeighbourField() was not faulty but it never got called
• instead it went through the two-step evaluation inside cyclicAMI which then caused messages to be sent but never consumed.
• yes, the new code solves the bug. The only problem with it is that it also disables the two-step evaluation which works great for parallelisation on massive numbers of cores.

added community label