foamCleanPath performs strangely if username/path has @ and . characters

Made the issue visible

By Luís Carvalheiro on 2016-10-10T15:28:08 (imported from GitLab project)

I'm guessing here, but I suspect that the problem has to do with the default shell /bin/sh that exists in the system that you're using. You can do a few checks to see if it's Bash or not, by running:

ls -l /bin/sh
ls -l /bin/bash

The two paths should either be of the same size or the first should be a symbolic link to the second one.

The other test can be this:

sh --version
bash --version

to see if they give the same exact version... although the previous command block should make it pretty clear already.

That is at least the first possibility. The second possibility could be due to how Bash is configured, namely if it's working in POSIX mode or not. Try editing the file bin/foamCleanPath and prefix the command export to the line that defines the wildcard variable, namely by changing this:

wildcard=$1

to this:

export wildcard=$1

In other words, the block you mentioned should now look like this:

# strip out wildcards via sed
while [ "$#" -ge 1 ]
do
    export wildcard=$1
    shift
    ##DEBUG echo "remove>$wildcard<" 1>&2
    if [ -n "$wildcard" ]
    then
        dirList=$(echo "$dirList:" | sed -e "s@${wildcard}[^:]*:@@g")
    fi
done

I expect this will work, because of the following commit on OpenFOAM-dev: https://github.com/OpenFOAM/OpenFOAM-dev/commit/0e429dad08a7157d7120b95a27f281772eee284b

By Bruno Santos on 2016-10-10T22:22:20 (imported from GitLab project)

Dear Bruno, Thanks for your kind feedback. Actually, the system's default shell is bash

-rwxr-xr-x 1 root root 960392 Ago 2 17:00 /bin/bash

GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I tried your approach, but the result is exactly the same as described earlier. FiY, i installed a CentOS7 virtual machine with the same configuration as our cluster (except for the usernames and home folder names) and OpenFoam compiles perfectly. Regards, luis

By Luís Carvalheiro on 2016-10-11T08:07:53 (imported from GitLab project)

Dear Luís,

Hold on, what about /bin/sh? Since you didn't provide the details for it, I'm still wondering if it's really the same one or not.

On the other hand, I've remembered about another possibility: the variables that are being cleaned by foamCleanPath might have unexpected characters that break the loop.

Before sourcing the OpenFOAM+ environment, you can check the variables that are cleaned up, by running:

echo $PATH
echo $LD_LIBRARY_PATH
echo $MANPATH

I'm guessing that either a space or an @ symbol is mixed in the path names.

Best regards, Bruno

By Bruno Santos on 2016-10-11T20:14:32 (imported from GitLab project)

Sorry, how did I not see the title of your report! OK, that explains it. The user name lcarvalheiro@UA.PT is indeed the main suspect. I'll have to do some testing, to try and figure this one out.

By Bruno Santos on 2016-10-11T20:16:45 (imported from GitLab project)

Bruno: Yes, you're right. the "@UA.PT" is the main culprit. by the way, sh in simlinked to bash and my paths are, by default (the intel compilers and intel mpi modules are enabled by default, although I can switch them off and use only the openmpi module)

echo $PATH 
/usr/lib64/qt-3.3/bin:/opt/intel/impi/4.1.3.048/intel64/bin:/opt/intel/itac/8.1.4.045/intel64/bin:/opt/intel/composer_xe_2013_sp1.2.144/bin/intel64:/opt/intel/composer_xe_2013_sp1.2.144/bin/intel64_mic:/opt/intel/ics/2013.1.046/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64/py26/bin:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gdb/intel64_mic/py26/bin:/opt/intel/composer_xe_2013_sp1.2.144/debugger/gui/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/bin/intel64:/home/lcarvalheiro@UA.PT/.local/bin:/home/lcarvalheiro@UA.PT/bin

echo $LD_LIBRARY_PATH 
/opt/intel/itac/8.1.4.045/mic/itac/slib_impi4:/opt/intel/itac/8.1.4.045/intel64/itac/slib_impi4:/opt/intel/impi/4.1.3.048/intel64/lib:/opt/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64/gcc4.4:/opt/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/opt/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64

echo $MANPATH 
/usr/share/man:/usr/local/share/man:/opt/intel/composer_xe_2013_sp1.2.144/man/en_US:/opt/intel/impi/4.1.3.048/man:/opt/intel/itac/8.1.4.045/man

Well...the foamCleanPath file clearly states, as notes:

this routine will fail when directories have embedded spaces

false matches possible if a wildcard contains '.' (sed regex)

the wildcards can themselves can be written together and separated by colons or whitespace

I'll have to check this further. Thanks for your kind help!

By Luís Carvalheiro on 2016-10-11T20:36:06 (imported from GitLab project)

I was very curious about this and I was starting to test this in a virtual machine with CentOS 7... but then I got stopped by the interactive user creation dialogue window, which does not allow me to create a user-name with "@" in its name. It only allows letters, digits, '.', '-' and '_'.

After circumventing it by creating a user with '-' instead of '@' and then editing /etc/passwd and renaming the home folder, I managed to test that the following change seems to work as intended, at least it did look like it had worked well enough:

# strip out wildcards via sed
while [ "$#" -ge 1 ]
do
    wildcard=$1
    shift
    ##DEBUG echo "remove>$wildcard<" 1>&2
    if [ -n "$wildcard" ]
    then
        dirList=$(echo "$dirList:" | sed -e "s|${wildcard}[^:]*:||g")
    fi
done

So essentially I replaced @ with |, which is a pretty much illegal character for file names, or at the very least it's fairly annoying to use it in a file/folder name.

As for the dot in "UA.PT", I'm not very worried about it... unless there is something else named like "UAxPT", which would be accounted as it being the same string.

Please let us know if this allows you to both build/install and run the OpenFOAM+ applications!

By Bruno Santos on 2016-10-11T23:02:31 (imported from GitLab project)

Dear Bruno:

following your suggestion, i replaced @ with | . I felt confident with that approach, since i share your point of view about illegal characters for file/folder names. I started it all by the very beginning: uncompressed the OpenFoam tgz, thirdparty tgz, and edited foamCleanPath according to your suggestion.

I ended up with the same problem described on my first post.

========================================
Done ThirdParty Allwmake
========================================

+ wmakePrintBuild -check
no git description found
+ wrmo OpenFOAM/global/global.o
wrmo error: could not find Make directory

I'll continue my explorations....(and counting on your kind efforts to help me!) Regards, L

By Luís Carvalheiro on 2016-10-12T08:26:33 (imported from GitLab project)

A quick update: i started from a clean path and defined everything in the PATH, LD_PATH, MAN_PATH by hand. I also defined WM_* variables by hand. (not a very elegant solution, but....)

Then, in src/Allwmake, i commented the line wmakePrintBuild -check || wrmo OpenFOAM/global/global.o

then i simply ran the top-level ./Allwmake. So far, so good...it's compiling...

EDIT: or not. just failed with

+ ./graphics/Allwmake
 error: could not find Make directory

By Luís Carvalheiro on 2016-10-12T12:31:16 (imported from GitLab project)

Well, I think I finally got it compiled. As it might be convenient for others struggling to get OpenFOAM-v1606+ compiled (facing the same difficulties I had), I'll try, as best as I can, to describe the changes I made.

First: cleaned my home folder ( from my PATH, LD_LYBRARY_PATH and MANPATH of my username (lcarvalheiro@UA.PT) manually. Unloaded both intel compilers and mpi modules, and loaded the openmpi module.

second, the hint from Bruno: in bin/foamCleanPath, near line 110, i replaced

# strip out wildcards via sed
while [ "$#" -ge 1 ]
do
    shift
    ##DEBUG echo "remove>$wildcard<" 1>&2
    if [ -n "$wildcard" ]
    then
        dirList=$(echo "$dirList:" | sed -e "s@${wildcard}[^:]*:@@g")
    fi
done

by

# strip out wildcards via sed
while [ "$#" -ge 1 ]
do
    wildcard=$1
    shift
    ##DEBUG echo "remove>$wildcard<" 1>&2
    if [ -n "$wildcard" ]
    then
        dirList=$(echo "$dirList:" | sed -e "s|${wildcard}[^:]*:||g")
    fi
done

Then, and since my installation directory is not the home folder and I don't want to write anything on there (my foamInstall variable points to something like /ON/OTHER/PLACE/$WM_PROJECT), i changed in etc/bashrc

# Location of user files
# ~~~~~~~~~~~~~~~~~~~~~~
export WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION

to

# Location of user files
# ~~~~~~~~~~~~~~~~~~~~~~
export WM_PROJECT_USER_DIR=$WM_PROJECT/$USER-$WM_PROJECT_VERSION

Finally, and according to OpenFOAM-plus#178 (closed) I used @Mattijs wmakeFunctions, replacing the original in /wmake/scripts

And it compiled. Some comments: I had some issues like

/usr/bin/ld: skipping incompatible /lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /lib/libdl.a when searching for -ldl
/usr/bin/ld: skipping incompatible /lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /lib/libm.a when searching for -lm
/usr/bin/ld: skipping incompatible /lib/libc.a when searching for -lc
/usr/bin/ld: skipping incompatible /lib/libc.a when searching for -lc

But when executing foamInstallationTest, I ended up with

Summary
-------------------------------------------------------------------------------
Base configuration ok.
Critical systems ok.

Done

I'm now running the tutorial cases, via ./Allrun in the tutorials directory. So far, so good...

Cheers.

Luis.

By Luís Carvalheiro on 2016-10-13T15:48:36 (imported from GitLab project)

Thanks Luis and Bruno for your comments.

@mark Could you please check this?

Mentioned in commit d0bbb4fa4d40aab175bcbc0daf2479606084866f

Mentioned in issue #277 (closed)

By Bruno Santos on 2016-10-24T22:18:47 (imported from GitLab project)

FYI: I've posted a patch for OpenFOAM v4 and dev here: http://bugs.openfoam.org/view.php?id=2306 I've used the pipe character instead of @mark's solution with the question-mark... :)

Regarding @lcarvalheiro's comment on commit d0bbb4fa4d40aab175bcbc0daf2479606084866f:

+ wmakePrintBuild -check
no git description found
+ wrmo OpenFOAM/global/global.o
wrmo error: could not find Make directory

this is solved in OpenFOAM v4 by changing this line in src/Allwmake:

wmakePrintBuild -check || wrmo OpenFOAM/global/global.o

to this:

wmakePrintBuild -check || wrmo OpenFOAM/global/global.o 2>/dev/null

@mark: This isn't in OpenFOAM-plus develop branch.

By Bruno Santos on 2016-10-24T23:34:58 (imported from GitLab project)

@lcarvalheiro: About the warnings ...skipping incompatible..., that sounds to me like there are some additional 32-bit binutils packages installed in your system. I haven't tried to reproduce the problem yet, but a clear location (from the output of running Allwmake) of where this happens would come in handy for reproducing it.

By Bruno Santos on 2016-10-24T23:37:41 (imported from GitLab project)

Hi Bruno, I don't think the 2>/dev/null should be necessary at all. Presuming of course that the updated wmakePrintBuild is being used (issue #174 (closed), commit caf145222460da62112a0543d636f3a7156bfd81). With this change, it must exit with a return code of 0 and the wrmo should never been seen.

To verify this, try the following:

wmakePrintBuild -check && echo "check is ok" || echo "needs rebuild"

If I move my OpenFOAM .git directory completely out of the way, this is what I receive.

no git description found
check is ok

If this is not what you receive, then the wmakePrintBuild on the system is an older one.

Hi Mark, many thanks for the reminder! I knew I had seen something about that fix, but I couldn't remember exactly where I saw it. That does make it a much cleaner solution than hiding the error.

By Bruno Santos on 2016-10-25T12:51:40 (imported from GitLab project)

Status changed to closed by commit d0bbb4fa4d40aab175bcbc0daf2479606084866f

By Andrew Heather on 2016-10-26T14:55:45 (imported from GitLab project)

mentioned in commit dad301d8

foamCleanPath performs strangely if username/path has @ and . characters

Child items

Activity

Admin message

foamCleanPath performs strangely if username/path has @ and . characters

Child items

Linked items

Activity