Illegal Operand when OpenFoam is compiled with GCC
Summary
I am compiling OpenFoam 2012 for the users of our Centos7 HPC systems using the spack recipe.
When I compile using Intel 19.1.1 everything runs as expected using the cavity tutorial as a test.
When I compile with gcc (I've tried 6.4 and 10.2) everything seems to work except when called with mpirun and --parallel. I have tried version 2012 and the current devel branch. The proximate error is "Illegal Operand" see below for the error stack.
Compiling OpenFoam-org version 7 with gcc 10.2.0 works with mpirun. One of our users has a particular need to use gcc and OpenFoam to write an extension.
Steps to reproduce
spack install openfoam%gcc@10.2
module load openfoam-develop-gcc-10.2.0-mkl-6r7rrzb
cd cavity
blockMesh
decomposePar
mpirun -np 2 -machinefile $PBS_NODEFILE icoFoam -parallel # Assuming a PBS allocated nodeWhat is the current bug behaviour?
[wheeler100:30014:0:30014] Caught signal 4 (Illegal instruction: illegal operand)
==== backtrace ====
[wheeler100:30013:0:30013] Caught signal 4 (Illegal instruction: illegal operand)Relevant logs and/or images
mfricke@wheeler100:~/test_programs/OpenFoam/cavity $ mpirun -np 2 -machinefile $PBS_NODEFILE icoFoam -parallel
[wheeler100:30014:0:30014] Caught signal 4 (Illegal instruction: illegal operand)
==== backtrace ====
[wheeler100:30013:0:30013] Caught signal 4 (Illegal instruction: illegal operand)
==== backtrace ====
0 /usr/lib64/libucs.so.0(+0x17970) [0x2aef18126970]
1 /usr/lib64/libucs.so.0(+0x179f3) [0x2aef181269f3]
2 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(+0x7f29f) [0x2aee4a7ef29f]
3 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(+0x11c45) [0x2aee4a781c45]
4 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(fi_getinfo+0x70d) [0x2aee4a78260d]
5 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(fi_getinfo+0x4c) [0x2aee4a786efc]
6 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(+0x872452) [0x2aee48fb2452]
7 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(+0x242b6e) [0x2aee48982b6e]
8 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(+0x579bd5) [0x2aee48cb9bd5]
9 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(PMPI_Init_thread+0x10b) [0x2aee48cba1d1]
10 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/openfoam-develop-6r7rrzbjrqxjejo6stnxarlrcwrsncr3/platforms/linux64GccDPInt32-spack/lib/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x82) [0x2aee47b6c3b2]
11 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/openfoam-develop-6r7rrzbjrqxjejo6stnxarlrcwrsncr3/platforms/linux64GccDPInt32-spack/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x767) [0x2aee460cc907]
12 icoFoam() [0x42244d]
13 /lib64/libc.so.6(__libc_start_main+0xf5) [0x2aee477b2555]
14 icoFoam() [0x426ac7]
===================
0 /usr/lib64/libucs.so.0(+0x17970) [0x2ab44ffa2970]
1 /usr/lib64/libucs.so.0(+0x179f3) [0x2ab44ffa29f3]
2 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(+0x7f29f) [0x2ab38266b29f]
3 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(+0x11c45) [0x2ab3825fdc45]
4 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(fi_getinfo+0x70d) [0x2ab3825fe60d]
5 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/libfabric-1.12.1-bgj3y4nzp2uuvkohviyxf5i5dit4ufzy/lib/libfabric.so.1(fi_getinfo+0x4c) [0x2ab382602efc]
6 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(+0x872452) [0x2ab380e2e452]
7 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(+0x242b6e) [0x2ab3807feb6e]
8 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(+0x579bd5) [0x2ab380b35bd5]
9 /opt/intel/compilers_and_libraries_2020.1.217/linux/mpi/intel64/lib/debug/libmpi.so.12(PMPI_Init_thread+0x10b) [0x2ab380b361d1]
10 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/openfoam-develop-6r7rrzbjrqxjejo6stnxarlrcwrsncr3/platforms/linux64GccDPInt32-spack/lib/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x82) [0x2ab37f9e83b2]
11 /opt/spack/opt/spack/linux-centos7-nehalem/gcc-10.2.0/openfoam-develop-6r7rrzbjrqxjejo6stnxarlrcwrsncr3/platforms/linux64GccDPInt32-spack/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x767) [0x2ab37df48907]
12 icoFoam() [0x42244d]
13 /lib64/libc.so.6(__libc_start_main+0xf5) [0x2ab37f62e555]
14 icoFoam() [0x426ac7]
===================
Here is the concretization from spack showing all the dependency versions:
mfricke@wheeler-sn:~/test_programs/OpenFoam/cavity $ spack spec openfoam
Input spec
--------------------------------
openfoam
Concretized
--------------------------------
openfoam@2012%gcc@10.2.0~float32~int64~kahip~knl~metis~mgridgen~paraview+scotch+source~spdp~vtk~zoltan arch=linux-centos7-nehalem
^adios2@2.7.1%gcc@10.2.0+blosc+bzip2~dataman~dataspaces~endian_reverse+fortran~hdf5~ipo+mpi+pic+png~python+shared+ssc+sst+sz+zfp build_type=Release arch=linux-centos7-nehalem
^bzip2@1.0.8%gcc@10.2.0~debug~pic+shared arch=linux-centos7-nehalem
^diffutils@3.7%gcc@10.2.0 arch=linux-centos7-nehalem
^libiconv@1.16%gcc@10.2.0 arch=linux-centos7-nehalem
^c-blosc@1.21.0%gcc@10.2.0+avx2~ipo build_type=RelWithDebInfo arch=linux-centos7-nehalem
^cmake@3.20.2%gcc@10.2.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=linux-centos7-nehalem
^ncurses@6.2%gcc@10.2.0~symlinks+termlib abi=none arch=linux-centos7-nehalem
^pkg-config@0.29.2%gcc@10.2.0+internal_glib arch=linux-centos7-nehalem
^openssl@1.1.1k%gcc@10.2.0~docs+systemcerts arch=linux-centos7-nehalem
^perl@5.32.1%gcc@10.2.0+cpanm+shared+threads arch=linux-centos7-nehalem
^berkeley-db@18.1.40%gcc@10.2.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=linux-centos7-nehalem
^gdbm@1.19%gcc@10.2.0 arch=linux-centos7-nehalem
^readline@8.1%gcc@10.2.0 arch=linux-centos7-nehalem
^zlib@1.2.11%gcc@10.2.0+optimize+pic+shared arch=linux-centos7-nehalem
^lz4@1.9.3%gcc@10.2.0 libs=shared,static arch=linux-centos7-nehalem
^snappy@1.1.8%gcc@10.2.0~ipo+pic+shared build_type=RelWithDebInfo patches=c9cfecb1f7a623418590cf4e00ae7d308d1c3faeb15046c2e5090e38221da7cd arch=linux-centos7-nehalem
^zstd@1.4.9%gcc@10.2.0~ipo~legacy~lz4~lzma+multithread+programs+shared+static~zlib build_type=RelWithDebInfo arch=linux-centos7-nehalem
^intel-mpi@2019.7.217%gcc@10.2.0 arch=linux-centos7-nehalem
^libfabric@1.12.1%gcc@10.2.0~kdreg fabrics=sockets,tcp,udp arch=linux-centos7-nehalem
^libffi@3.3%gcc@10.2.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=linux-centos7-nehalem
^libpng@1.6.37%gcc@10.2.0 arch=linux-centos7-nehalem
^sz@2.1.11.1%gcc@10.2.0~fortran~hdf5~ipo~netcdf~pastri~python~random_access+shared~stats~time_compression build_type=RelWithDebInfo arch=linux-centos7-nehalem
^zfp@0.5.5%gcc@10.2.0~aligned~c~cuda~fasthash~fortran~ipo~openmp~profile~python+shared~strided~twoway bsws=64 build_type=RelWithDebInfo cuda_arch=none arch=linux-centos7-nehalem
^boost@1.76.0%gcc@10.2.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 visibility=hidden arch=linux-centos7-nehalem
^cgal@4.13%gcc@10.2.0~core~demos+eigen~header_only~imageio~ipo+shared build_type=Release arch=linux-centos7-nehalem
^eigen@3.3.9%gcc@10.2.0~ipo build_type=RelWithDebInfo arch=linux-centos7-nehalem
^gmp@6.2.1%gcc@10.2.0 arch=linux-centos7-nehalem
^autoconf@2.69%gcc@10.2.0 arch=linux-centos7-nehalem
^m4@1.4.18%gcc@10.2.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-centos7-nehalem
^libsigsegv@2.13%gcc@10.2.0 arch=linux-centos7-nehalem
^automake@1.16.3%gcc@10.2.0 arch=linux-centos7-nehalem
^libtool@2.4.6%gcc@10.2.0 arch=linux-centos7-nehalem
^mpfr@4.1.0%gcc@10.2.0 arch=linux-centos7-nehalem
^autoconf-archive@2019.01.06%gcc@10.2.0 arch=linux-centos7-nehalem
^texinfo@6.5%gcc@10.2.0 patches=12f6edb0c6b270b8c8dba2ce17998c580db01182d871ee32b7b6e4129bd1d23a,1732115f651cff98989cb0215d8f64da5e0f7911ebf0c13b064920f088f2ffe1 arch=linux-centos7-nehalem
^flex@2.6.4%gcc@10.2.0+lex~nls patches=09c22e5c6fef327d3e48eb23f0d610dcd3a35ab9207f12e0f875701c677978d3 arch=linux-centos7-nehalem
^bison@3.7.6%gcc@10.2.0 arch=linux-centos7-nehalem
^findutils@4.8.0%gcc@10.2.0 arch=linux-centos7-nehalem
^gettext@0.21%gcc@10.2.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos7-nehalem
^libxml2@2.9.10%gcc@10.2.0~python arch=linux-centos7-nehalem
^xz@5.2.5%gcc@10.2.0~pic libs=shared,static arch=linux-centos7-nehalem
^tar@1.34%gcc@10.2.0 arch=linux-centos7-nehalem
^help2man@1.47.16%gcc@10.2.0 arch=linux-centos7-nehalem
^intel-mkl@2020.1.217%gcc@10.2.0~ilp64+shared threads=none arch=linux-centos7-nehalem
^scotch@6.0.10%gcc@10.2.0+compression~esmumps~int64~metis+mpi+shared arch=linux-centos7-nehalem
Environment information
- OpenFOAM version : 2021 and develop
- Operating system : CentOS Linux release 7.9.2009 (Core)
- Hardware info : Intel(R) Xeon(R) CPU X5550 @ 2.67GHz, 48 GB RAM, Infiniband ConnectX-4
- Compiler : GCC 10.2.0 and GCC 6.4.0
Any help would be appreciated,
Matthew
Edited by George Fricke