Different results with interFoam for serial and parallel simulations with fixed time step

Summary

The parallel execution of a two-phase-simulation with interFoam results in a partly very different solution for the flow fields compared to a serial and to other distributed simulations. The simulation consists of a domain initially filled with air. Over time the domain is filled by a water jet from the top. The air leaves the domain at an outlet located on the side. The following figure gives an overview.

The differences are observed for both, metis and simple, decomposition methods. Furthermore, there are also different results with different number of processors (e.g. between 4 and 8).

We have also uploaded the complete case to github.

Steps to reproduce

Clone this github and run simulation on different number of processes

Example case

See here

What is the current bug behaviour?

Results are not equal for different decompositions and also not equal to the serial case. The following figure gives the result of a simulation on 4 (top left) and 8 (top right) processes. Bottom pictures give the result, again 4 (bottom left) and 8 (bottom right) processes, with a limited Courant number.

The above figures show field U on a plane located in the center in z direction of the case. Next figure shows the location of the plane.

What is the expected correct behavior?

The results of different decomposed simulations and the serial simulation should be identical.

Relevant logs and/or images

Environment information

• OpenFOAM version : OpenFOAM-v2112 14aeaf8d
• Operating system : openSUSE Leap 15.3
• Hardware info : -
• Compiler : gcc (SUSE Linux) 7.5.0

Possible fixes

The limitation (see this diff) of the Courant number to a value of 1 gives much more similar results. Comare again bottom pictures within figure above.

added community label

assigned to @kuti

Hi @atismer - many thanks for the ticket.

Could you please write out the sequence of commands you executed for serial and parallel cases?

Thanks for the fast reply.

And yes, I can. The commands are very simple.

For the parallel case:

decomposePar
mpirun -np 4 interFoam -parallel

For the serial case:

interFoam

Should I also provide the log files of the simulations?

no, not yet. thanks. I'll be back next mid-week, and reproduce the problem.

changed the description

changed the description

Update description

• Add revision id of OpenFOAM Version
• Add position of evaluation plane

Hi did you find the solution to this, i am also having the same problem,

No, not yet. The only fix is the above mentioned limitation of the Co-number.

Does #2783 (closed) help? Do you see the issue in the

• feature-evaluation-check
• feature-evaluation-check-bugfix-only

branches?

Yes, I still see the issue in feature-evaluation-check. The solutions on different number of processors are not equal. The figure shows the result at 0.008s for 4 (left) and 8 (right) processors.

Should I still test with feature-evaluation-check-bugfix-only?

No. We did some testing internally and the problem seems to go away (or get a lot less severe) with a parallel consistent solver (PCG with diagonal preconditioning) or tighter solver tolerances.

Can you provide the modified files? Or can you explain a little bit more in detail the changes of the solution process?

in your fvSolution:

    solver          PCG;
    preconditioner  diagonal;

The diagonally-preconditioned-PCG should behave the same independent of decomposition. All the other preconditioners use neighbouring cell information. The neighbouring cell information is lagged in case of processor boundaries. Try running e.g. cavity case with scotch or random decomposition to see the effect.

Thank you very much. I test the recommended configuration. See figure:

But as you already mentioned:

or get a lot less severe

The results are still different, when not fixing the Co number.