inconsistency in scotch, metis library locations
Scotch and metis libraries are being installed under FOAM_EXT_LIBBIN, not under their respective XX_ARCH_PATH/lib. If there are copies in the XX_ARCH_PATH/lib, they will not be available in the LD_LIBRARY_PATH anyhow and thus is confusing (wrong) to have them as LIB_LIB.
Activity
I've taken a look at the commit and I have to say that I'm a bit confused... How are system-wide installations of Metis and Scotch meant to be supported?
I know that Scotch has been lagging behind on a few Linux Distributions (e.g. on Debian), but Metis is being used by a few people + installation instructions to rely on the system's installation, e.g. on openSUSE.
Just the other day there was a questions about this, namely for Scotch: http://www.cfd-online.com/Forums/openfoam-installation/178828-install-openfoam-using-system-scotch.html - it's a bit outdated, because the forum member was asking for OpenFOAM 2.4.0, but still...
By Bruno Santos on 2016-11-04T21:30:21 (imported from GitLab project)
Hi Bruno, I see your point. None of the configurations (previous and current) have provision for system installations (ie, via RPM or equivalent). This sounds reasonable to support. Probably should also support central non-system, non-thirdparty installations similar to our boost/cgal configuration.
For the rest of the commit, the basic problem was that we had third-party metis being installed into METIS_ARCH_PATH/lib and then being copied to the FOAM_EXT_LIBBIN path, which meant that there could be a difference in the linked library and the run-time library.
Hi Mark,
None of the configurations (previous and current) have provision for system installations (ie, via RPM or equivalent). This sounds reasonable to support.
There were several hack'ish ways that were being used for supporting system installations, each implemented back when they were first needed and occasionally adjusted as time went by...
And I was going to point out such a situation where I had used the existing framework to use the system's Metis on openSUSE: https://openfoamwiki.net/index.php/Installation/Linux/OpenFOAM-4.1/openSUSE#openSUSE_Leap_42.1 - but then noticed the issue about using
lib64, which I didn't notice when I wrote the instructions... anyway, I've updated the instructions and it will only work for 64-bit labels. The personal/custom setting used and hack used is in step #6 (closed), namely:mkdir -p ~/.OpenFOAM/4.1/config.sh echo export METIS_VERSION=metis-5.1.0 > ~/.OpenFOAM/4.1/config.sh/metis echo export METIS_ARCH_PATH=/usr >> ~/.OpenFOAM/4.1/config.sh/metis sed -i -e 's=/lib\s=/lib64 =' OpenFOAM-4.1/src/parallel/decompose/metisDecomp/Make/options sed -i -e 's=/lib/=/lib64/=' OpenFOAM-4.1/src/parallel/decompose/metisDecomp/AllwmakeProbably should also support central non-system, non-thirdparty installations similar to our boost/cgal configuration.
Yes and that would make it consistent and on-par with another report that is open somewhere around here... ah, here we go, issue #130 (closed) :)
For the rest of the commit, the basic problem was that we had third-party metis being installed into METIS_ARCH_PATH/lib and then being copied to the FOAM_EXT_LIBBIN path, which meant that there could be a difference in the linked library and the run-time library.
I don't remember noticing that before. This is rather ambiguous, although I guess it was a workaround that worked well back when it was first implemented... I now see why you were relying on
FOAM_EXT_LIBBIN! Many thanks!By Bruno Santos on 2016-11-06T16:15:19 (imported from GitLab project)
-
@wyldckat Added support for system installation and site-wide installation, with sanity checks on the sizes (ie, OpenFOAM vs metis o scotch header). If you want to test for any remaining trouble, this would be very welcome.
@mark That is indeed a lot of sanity checks! I have to say that it feels very awkward checking for the sizes through shell scripting... I'll try to find some time to double-check the changes you've made, but I can't guarantee that I'll be able to do it before v1612+ is released.
By the way, this topic is somewhat being discussed on the OpenFOAM Foundation tracker: http://bugs.openfoam.org/view.php?id=2330 - one of the interesting details discussed there is regarding the support for conversion between integers, e.g. on openSUSE Metis is built with 64-bit integers, but OpenFOAM is usually built with 32-bit integers, therefore it makes some sense to support this interface... at least from a system's package management perspective.
By Bruno Santos on 2016-11-13T13:38:12 (imported from GitLab project)
-
@wyldckat I did stumble across the proposed patch. The check for library with a
.../lib*/libmetis.sois fairly weird. I hope no one actually does that. I agree that the change for processor weights to float is probably quite reasonable. The number of processors is relatively small, so the overhead is minimal and the overall impact is not too bad.I'm still undecided about the treatment of integers. The packaging argument certainly makes sense, but would still need a sanity check that you do not mix 32-bit metis and 64-bit OpenFOAM labels - since this would spell certain disaster. In the other direction, you potentially have lots of copying about (going from 32-bit foam to 64-bit metis labelling). But even if decomposition is within a solver, it would normally occur before any calculations and thus the overall memory peak should still be fine.
The check for library with a .../lib*/libmetis.so is fairly weird.
That reminded me just now that in some situations, architecture specific libraries are actually placed at
/usr/lib/$(uname -a)-linux-gnu, not inlib64... which is usually done by package managers, e.g. VTK 6.2 libraries on Ubuntu 16.04.
By the way, the following in "src/parallel/decompose/Allwmake" (small excerpt):
s/^[ tab]*#[ tab]does not work for replacing tabs. The following does (respective excerpt) for both spaces and tabs:
s/^[\s\t]*#[\s\t]By Bruno Santos on 2016-11-13T15:19:58 (imported from GitLab project)
