mapFieldsPar problem when running in Parallel
Summary
When executing the mapFieldsPar in parallel trying to map one box with 2 elements into another box of 2 or 4, or more elements I encounter a out of index error. When executing the same in serial it works fine.
Steps to reproduce
In the target source I execute the following:
mpirun -np 2 mapFieldsPar /decayIsoTurb4.1_copyWithLessTime -sourceTime latestTime -parallel
What is the current bug behaviour?
/*---------------------------------------------------------------------------*\
| ========= | |
| \\ / F ield | OpenFOAM: The Open Source CFD Toolbox |
| \\ / O peration | Version: v2006 |
| \\ / A nd | Website: www.openfoam.com |
| \\/ M anipulation | |
\*---------------------------------------------------------------------------*/
Build : 5e660c36e9-20201103 OPENFOAM=2006 patch=201012
Arch : "LSB;label=32;scalar=64"
Exec : mapFieldsPar /home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric/decayIsoTurb4.1_copyWithLessTime -sourceTime latestTime -parallel
Date : Aug 07 2021
Time : 10:09:18
Host : pablo-XPS-13-9370
PID : 13775
I/O : uncollated
Case : /home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric/decayIsoTurb4.1_trialWithLessElements
nProcs : 2
Hosts :
(
(pablo-XPS-13-9370 2)
)
Pstream initialized with:
floatTransfer : 0
nProcsSimpleSum : 0
commsType : nonBlocking
polling iterations : 0
trapFpe: Floating point exception trapping enabled (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster (fileModificationSkew 5, maxFileModificationPolls 20)
allowSystemOperations : Allowing user-supplied system call operations
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Source: "/home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric" "decayIsoTurb4.1_copyWithLessTime"
Target: "/home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric" "decayIsoTurb4.1_trialWithLessElements"
Create databases as time
I/O : uncollated
Case : /home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric/decayIsoTurb4.1_copyWithLessTime
nProcs : 2
Hosts :
(
(pablo-XPS-13-9370 2)
)
Pstream initialized with:
floatTransfer : 0
nProcsSimpleSum : 0
commsType : nonBlocking
polling iterations : 0
Overriding DebugSwitches according to controlDict
meshToMesh 2;
Source time: 0
Target time: 0
Create meshes
Source mesh size: 2 Target mesh size: 2
Creating and mapping fields for time 0
Creating mesh-to-mesh addressing for region0 and region0 regions using cellVolumeWeight
From Foam::label Foam::meshToMesh::calcDistribution(const Foam::polyMesh&, const Foam::polyMesh&) const
in file meshToMesh/meshToMeshParallelOps.C at line 70
Meshes split across multiple processors
meshToMesh: Using AABBTree method
From Foam::autoPtr<Foam::mapDistribute> Foam::meshToMesh::calcProcMap(const Foam::polyMesh&, const Foam::polyMesh&) const
in file meshToMesh/meshToMeshParallelOps.C at line 176
Determining extent of src mesh per processor:
proc bb
0 2{(-0.0003 -0.0003 -0.0003) (0.0203 0.0203 0.0103)}
1 2{(-0.0003 -0.0003 0.0097) (0.0203 0.0203 0.0203)}
[0] Of my 1 target cells I need to send to:
[0] proc cells
[0] 0 1
[0] 1 1
[1] Of my 1 target cells I need to send to:
[1] proc cells
[1] 0 1
[1] 1 1
[0] tgtProc:0 sending tgt cell 0[0] to srcProc 0
[1] tgtProc:1 sending tgt cell 0[1] to srcProc 0
[0] tgtProc:0 sending tgt cell 0[0] to srcProc 1
[1] tgtProc:1 sending tgt cell 0[1] to srcProc 1
[0] Target mesh send sizes[0]: points=8, faces=6, nInternalFaces=0, faceOwn=6, faceNbr=6, cellIDs=1
[0] Target mesh send sizes[1]: points=8, faces=6, nInternalFaces=0, faceOwn=6, faceNbr=6, cellIDs=1
[1] Target mesh send sizes[0]: points=8, faces=6, nInternalFaces=0, faceOwn=6, faceNbr=6, cellIDs=1
[1] Target mesh send sizes[1]: points=8, faces=6, nInternalFaces=0, faceOwn=6, faceNbr=6, cellIDs=1
[0] Additional internal face between procs:0 and 1 across local face 0
[0] Additional internal face between procs:0 and 1 across local face 0
[1] Additional internal face between procs:0 and 1 across local face 0
[1] Additional internal face between procs:0 and 1 across local face 0
[0] Additional internal face between procs:0 and 1 across local face 0
[1] Additional internal face between procs:0 and 1 across local face 0
[0] proc 0 inserting face:0 connection between owner 0 and neighbour 0
[0] proc 1 inserting face:1 connection between owner 1 and neighbour -2
[1] proc 0 inserting face:0 connection between owner 0 and neighbour 0
[1] proc 1 inserting face:1 connection between owner 1 and neighbour -2
[1] proc 1 inserting face:2 connection between owner 1 and neighbour -2
[0] proc 1 inserting face:2 connection between owner 1 and neighbour -2
[0]
[0]
[0] --> FOAM FATAL ERROR:
[0] index 9 out of range [0,9]
[0]
[0] From void Foam::UList<T>::checkIndex(Foam::label) const [with T = Foam::face; Foam::label = int]
[0] in file /home/pablo/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/UListI.H at line 138.
[0]
FOAM parallel run aborting
[0]
[1]
[1]
[1] --> FOAM FATAL ERROR:
[1] index 9 out of range [0,9]
[1]
[1] From void Foam::UList<T>::checkIndex(Foam::label) const [with T = Foam::face; Foam::label = int]
[1] in file /home/pablo/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/UListI.H at line 138.
[1]
FOAM parallel run aborting
[0]
[1]
[1]
[1] --> FOAM FATAL ERROR:
[1] index 9 out of range [0,9]
[1]
[1] From void Foam::UList<T>::checkIndex(Foam::label) const [with T = Foam::face; Foam::label = int]
[1] in file /home/pablo/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/UListI.H at line 138.
[1]
[1]
[0] #0 [1] #0 Foam::error::printStack(Foam::Ostream&)Foam::error::printStack(Foam::Ostream&) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OSspecific/POSIX/printStack/printStack.C:237
at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OSspecific/POSIX/printStack/printStack.C:237
[1] #1 Foam::error::exitOrAbort(int, bool)[0] #1 Foam::error::exitOrAbort(int, bool) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/error.C:265
at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/error.C:265
[1] #2 Foam::error::abort()[0] #2 Foam::error::abort() at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/error.C:307
[0] #3 at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/error.C:307
[1] #3 Foam::Ostream& Foam::operator<< <Foam::error>(Foam::Ostream&, Foam::errorManip<Foam::error>)Foam::Ostream& Foam::operator<< <Foam::error>(Foam::Ostream&, Foam::errorManip<Foam::error>) in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[0] #4 in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[1] #4 Foam::UList<Foam::face>::checkIndex(int) constFoam::UList<Foam::face>::checkIndex(int) const in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[0] #5 Foam::UList<Foam::face>::operator[](int) in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[1] #5 Foam::UList<Foam::face>::operator[](int) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/UListI.H:251
at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/OpenFOAM/lnInclude/UListI.H:251
[0] #6 Foam::meshToMesh::distributeAndMergeCells(Foam::mapDistribute const&, Foam::polyMesh const&, Foam::globalIndex const&, Foam::Field<Foam::Vector<double> >&, Foam::List<Foam::face>&, Foam::List<int>&, Foam::List<int>&, Foam::List<int>&) const[1] #6 Foam::meshToMesh::distributeAndMergeCells(Foam::mapDistribute const&, Foam::polyMesh const&, Foam::globalIndex const&, Foam::Field<Foam::Vector<double> >&, Foam::List<Foam::face>&, Foam::List<int>&, Foam::List<int>&, Foam::List<int>&) const at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMeshParallelOps.C:860 (discriminator 1)
[0] #7 Foam::meshToMesh::calculate(Foam::word const&, bool) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMeshParallelOps.C:860 (discriminator 1)
[1] #7 Foam::meshToMesh::calculate(Foam::word const&, bool) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMesh.C:466 (discriminator 1)
[1] #8 Foam::meshToMesh::constructFromCuttingPatches(Foam::word const&, Foam::word const&, Foam::HashTable<Foam::word, Foam::word, Foam::string::hash> const&, Foam::List<Foam::word> const&, bool) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMesh.C:466 (discriminator 1)
[0] #8 Foam::meshToMesh::constructFromCuttingPatches(Foam::word const&, Foam::word const&, Foam::HashTable<Foam::word, Foam::word, Foam::string::hash> const&, Foam::List<Foam::word> const&, bool) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMesh.C:832
at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMesh.C:832
[1] #9 Foam::meshToMesh::meshToMesh(Foam::polyMesh const&, Foam::polyMesh const&, Foam::word const&, Foam::word const&, Foam::HashTable<Foam::word, Foam::word, Foam::string::hash> const&, Foam::List<Foam::word> const&, Foam::meshToMesh::procMapMethod const&, bool)[0] #9 Foam::meshToMesh::meshToMesh(Foam::polyMesh const&, Foam::polyMesh const&, Foam::word const&, Foam::word const&, Foam::HashTable<Foam::word, Foam::word, Foam::string::hash> const&, Foam::List<Foam::word> const&, Foam::meshToMesh::procMapMethod const&, bool) at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMesh.C:980
[0] #10 at ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/src/sampling/meshToMesh/meshToMesh.C:980
[1] #10 ?? in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[0] #11 in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[1] #11 ?? in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[0] #12 __libc_start_main in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsPar
[1] #12 __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
[0] #13 in /lib/x86_64-linux-gnu/libc.so.6
[1] #13 ?? in ~/BuildOpenFoamPlus/OpenFoamGit/openfoam/platforms/linux64Gcc75DPInt32Debug/bin/mapFieldsParWhat is the expected correct behavior?
When running in serial the result is as expected:
mapFieldsPar /decayIsoTurb4.1_copyWithLessTime -sourceTime latestTimeSource: "/home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric" "decayIsoTurb4.1_copyWithLessTime"
Target: "/home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric" "decayIsoTurb4.1_trialWithLessElements"
Create databases as time
I/O : uncollated
Case : /home/pablo/Desktop/ApplicationRuhr/Data/paperIlana/twoPointCorrEnric/decayIsoTurb4.1_copyWithLessTime
nProcs : 1
Overriding DebugSwitches according to controlDict
meshToMesh 2;
Source time: 0
Target time: 0
Create meshes
Source mesh size: 2 Target mesh size: 2
Creating and mapping fields for time 0
Creating mesh-to-mesh addressing for region0 and region0 regions using cellVolumeWeight
Source size = 2
Target size = 2
Overlap volume: 8e-06
interpolating onto existing field U
ExecutionTime = 0.03 s ClockTime = 0 s
EndEnvironment information
-
OpenFOAM version : v2006 and v2012
-
Operating system : Ubuntu 18.04 and OpenSuse 15.2
-
Hardware info :
-
Compiler : gcc7.5
The "/label bug" text is a gitlab flag that will add the "bug" label to this issue -->