Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
Development
openfoam
Commits
b5dddd89
Commit
b5dddd89
authored
Nov 16, 2010
by
mattijs
Browse files
ENH: parallel runTimeModifiable - master only
parent
57a443a1
Changes
13
Expand all
Hide whitespace changes
Inline
Side-by-side
doc/changes/inotify.txt
View file @
b5dddd89
...
...
@@ -24,21 +24,63 @@ be quite a gain on large numbers of processors.
- all file monitoring is done by an instance of 'fileMonitor' in the Time
class. The fileMonitor class can be found in OSspecific. Default is
to use the (linux-specific) 'inotify' system calls.
If compiled with -DFOAM_USE_STAT it will revert to the current 'stat' system
calls.
class. The fileMonitor class can be found in OSspecific. It uses either
timestamps as before or the (linux-specific) 'inotify' system framework
(available only if compiled with -DFOAM_USE_INOTIFY).
- inotify does not need timestamps. There is no need for fileModificationSkew
to allow for time differences. (there can still temporarily be a difference
in modified status between different processors due to nfs lagging)
- the monitoring can be done in one of four modes as set by
OptimisationSwitches::fileModificationChecking
- timeStamp : old behaviour : all nodes check the timestamp
- inotify : using inotify instead of timestamps
- timeStampMaster,inotifyMaster : only the master node checks the file
and only the master node reads it and distribute it to the
slaves. This makes runTimeModifiable possible on distributed
running (see below).
- distributed running:
- set fileModificationChecking to e.g. timeStampMaster
- decompose a case, e.g. cavity
- copy system and constant to processor0/
- put the all the processor* directories on the wanted nodes inside
the case directory. E.g.
- on master have /tmp/cavity/processor0
- on slaveN have /tmp/cavity/processorN
- so to reiterate:
- there is no need for cavity/constant or cavity/system, all the
dictionaries are only in processor0/constant or processor0/system
- the slave processor directories have no system directory and the
constant directory only contains the mesh.
- start the job in distributed mode by specifying the slave roots
(so one less than the number of processors) with
the -roots command line option:
mpirun -np 2 icoFoam -roots '("/tmp")' -parallel
- the alternative to the -roots option is to have a
cavity/system/decomposeParDict on the master with
distributed yes;
roots ("/tmp");
Details:
- timeStampMaster, inotifyMaster : this works only for IOdictionaries that
are READ_IF_MODIFIED. It means that slaves read exactly the same dictionary
as the master so cannot be used for dictionaries that contain e.g. mesh
specific information.
- inotify is a monitoring framework used to monitor changes in
lots of files (e.g. used in desktop searched like beagle). You specify
files to monitor and then get warned for any changes to these files.
It does not need timestamps. There is no need for fileModificationSkew
to allow for time differences. (there can still temporarily be a difference
in modified status between different processors due to nfs lagging). The big
problem is that it does not work over nfs3 (not sure about nfs4).
- fileMonitor stores two hashtables per file so there is a small overhead
adding and removing files from monitoring.
- if runTimeModifiable is false at start of run no files will get monitored,
however if runTimeModified gets set to false during the run the files
will still get monitored (though never reloaded). This is only a hypothetical
...
...
@@ -46,7 +88,6 @@ problem in that the kernel still stores events for the monitored files. However
inotify is very efficient - e.g. it gets used to track changes on file systems
for desktop search engines.
- in the old system one could call modified() on any object and get
and uptodate state. In the new system it will return the state from
the last runTime++ (which if it triggered any re-reads will have reset the
...
...
etc/controlDict
View file @
b5dddd89
...
...
@@ -872,6 +872,14 @@ InfoSwitches
OptimisationSwitches
{
fileModificationSkew 10;
//- Modification checking:
// - timeStamp : use modification time on file
// - inotify : use inotify framework
// - timeStampMaster : do time stamp (and file reading) only on master.
// - inotifyMaster : do inotify (and file reading) only on master.
fileModificationChecking timeStampMaster;//inotify;timeStamp;inotifyMaster;
commsType nonBlocking; //scheduled; //blocking;
floatTransfer 0;
nProcsSimpleSum 0;
...
...
src/OSspecific/POSIX/Allwmake
View file @
b5dddd89
...
...
@@ -12,9 +12,9 @@ unset COMP_FLAGS LINK_FLAGS
if
[
-f
/usr/include/sys/inotify.h
-a
"
${
1
%USE_STAT
}
"
=
"
$1
"
]
then
echo
"Found <sys/inotify.h> -- using inotify for file monitoring."
unse
t
COMP_FLAGS
expor
t
COMP_FLAGS
=
"-DFOAM_USE_INOTIFY"
else
expor
t
COMP_FLAGS
=
"-DFOAM_USE_STAT"
unse
t
COMP_FLAGS
fi
...
...
src/OSspecific/POSIX/fileMonitor.C
View file @
b5dddd89
This diff is collapsed.
Click to expand it.
src/OSspecific/POSIX/fileMonitor.H
View file @
b5dddd89
...
...
@@ -28,10 +28,11 @@ Description
Checking for changes to files.
Note
The default is to use
inotify (Linux specific, since 2.6.13)
The default is to use
stat to get the timestamp.
Compiling with FOAM_USE_STAT (or if /usr/include/sys/inotify.h
does not exist) uses the stat function call.
Compile with FOAM_USE_INOTIFY to use the inotify
(Linux specific, since 2.6.13) framework. The problem is that inotify does
not work on nfs3 mounted directories!!
SourceFiles
fileMonitor.C
...
...
@@ -78,6 +79,9 @@ public:
private:
// Private data
//- Whether to use inotify (requires -DFOAM_USE_INOTIFY, see above)
const
bool
useInotify_
;
//- State for all watchFds
mutable
DynamicList
<
fileState
>
state_
;
...
...
@@ -111,7 +115,7 @@ public:
// Constructors
//- Construct null
fileMonitor
();
fileMonitor
(
const
bool
useInotify
);
//- Destructor
...
...
@@ -133,7 +137,11 @@ public:
fileState
getState
(
const
label
watchFd
)
const
;
//- Check state of all files. Updates state_.
void
updateStates
(
const
bool
syncPar
)
const
;
void
updateStates
(
const
bool
masterOnly
,
const
bool
syncPar
)
const
;
//- Reset state (e.g. after having read it) using handle
void
setUnmodified
(
const
label
watchFd
);
...
...
src/OpenFOAM/db/IOobjects/IOdictionary/IOdictionary.C
View file @
b5dddd89
...
...
@@ -31,12 +31,61 @@ Description
#include
"IOdictionary.H"
#include
"objectRegistry.H"
#include
"Pstream.H"
// * * * * * * * * * * * * * * Static Data Members * * * * * * * * * * * * * //
defineTypeNameAndDebug
(
Foam
::
IOdictionary
,
0
);
// * * * * * * * * * * * * * Private Member Functions * * * * * * * * * * * //
// Parallel aware reading, using non-virtual type information (typeName instead
// of type()) because of use in constructor.
void
Foam
::
IOdictionary
::
readFile
(
const
bool
masterOnly
)
{
if
(
Pstream
::
master
()
||
!
masterOnly
)
{
if
(
debug
)
{
Pout
<<
"IOdictionary : Reading "
<<
objectPath
()
<<
" from file "
<<
endl
;
}
readStream
(
typeName
)
>>
*
this
;
close
();
}
if
(
masterOnly
)
{
// Scatter master data
if
(
Pstream
::
master
())
{
for
(
int
slave
=
Pstream
::
firstSlave
();
slave
<=
Pstream
::
lastSlave
();
slave
++
)
{
OPstream
toSlave
(
Pstream
::
scheduled
,
slave
);
IOdictionary
::
writeData
(
toSlave
);
}
}
else
{
if
(
debug
)
{
Pout
<<
"IOdictionary : Reading "
<<
objectPath
()
<<
" from master processor "
<<
Pstream
::
masterNo
()
<<
endl
;
}
IPstream
fromMaster
(
Pstream
::
scheduled
,
Pstream
::
masterNo
());
IOdictionary
::
readData
(
fromMaster
);
}
}
}
// * * * * * * * * * * * * * * * * Constructors * * * * * * * * * * * * * * //
Foam
::
IOdictionary
::
IOdictionary
(
const
IOobject
&
io
)
...
...
@@ -56,17 +105,41 @@ Foam::IOdictionary::IOdictionary(const IOobject& io)
//<< abort(FatalError);
}
// Everyone check or just master
bool
masterOnly
=
regIOobject
::
fileModificationChecking
==
timeStampMaster
||
regIOobject
::
fileModificationChecking
==
inotifyMaster
;
// Check if header is ok for READ_IF_PRESENT
bool
isHeaderOk
=
false
;
if
(
io
.
readOpt
()
==
IOobject
::
READ_IF_PRESENT
)
{
if
(
masterOnly
)
{
if
(
Pstream
::
master
())
{
isHeaderOk
=
headerOk
();
}
Pstream
::
scatter
(
isHeaderOk
);
}
else
{
isHeaderOk
=
headerOk
();
}
}
if
(
(
io
.
readOpt
()
==
IOobject
::
MUST_READ
||
io
.
readOpt
()
==
IOobject
::
MUST_READ_IF_MODIFIED
)
||
(
io
.
readOpt
()
==
IOobject
::
READ_IF_PRESENT
&&
h
eaderOk
())
||
isH
eaderOk
)
{
readStream
(
typeName
)
>>
*
this
;
close
();
readFile
(
masterOnly
);
}
dictionary
::
name
()
=
IOobject
::
objectPath
();
...
...
@@ -90,17 +163,41 @@ Foam::IOdictionary::IOdictionary(const IOobject& io, const dictionary& dict)
<<
endl
;
}
// Everyone check or just master
bool
masterOnly
=
regIOobject
::
fileModificationChecking
==
timeStampMaster
||
regIOobject
::
fileModificationChecking
==
inotifyMaster
;
// Check if header is ok for READ_IF_PRESENT
bool
isHeaderOk
=
false
;
if
(
io
.
readOpt
()
==
IOobject
::
READ_IF_PRESENT
)
{
if
(
masterOnly
)
{
if
(
Pstream
::
master
())
{
isHeaderOk
=
headerOk
();
}
Pstream
::
scatter
(
isHeaderOk
);
}
else
{
isHeaderOk
=
headerOk
();
}
}
if
(
(
io
.
readOpt
()
==
IOobject
::
MUST_READ
||
io
.
readOpt
()
==
IOobject
::
MUST_READ_IF_MODIFIED
)
||
(
io
.
readOpt
()
==
IOobject
::
READ_IF_PRESENT
&&
h
eaderOk
())
||
isH
eaderOk
)
{
readStream
(
typeName
)
>>
*
this
;
close
();
readFile
(
masterOnly
);
}
else
{
...
...
src/OpenFOAM/db/IOobjects/IOdictionary/IOdictionary.H
View file @
b5dddd89
...
...
@@ -57,6 +57,11 @@ class IOdictionary
public
dictionary
{
// Private Member Functions
//- read dictionary from file
void
readFile
(
const
bool
);
public:
TypeName
(
"dictionary"
);
...
...
src/OpenFOAM/db/Time/Time.C
View file @
b5dddd89
...
...
@@ -250,8 +250,27 @@ Foam::Time::Time
// Time objects not registered so do like objectRegistry::checkIn ourselves.
if
(
runTimeModifiable_
)
{
monitorPtr_
.
reset
(
new
fileMonitor
());
controlDict_
.
watchIndex
()
=
addWatch
(
controlDict_
.
filePath
());
monitorPtr_
.
reset
(
new
fileMonitor
(
regIOobject
::
fileModificationChecking
==
inotify
||
regIOobject
::
fileModificationChecking
==
inotifyMaster
)
);
// File might not exist yet.
fileName
f
(
controlDict_
.
filePath
());
if
(
!
f
.
size
())
{
// We don't have this file but would like to re-read it.
// Possibly if in master-only reading mode. Use a non-existing
// file to keep fileMonitor synced.
f
=
controlDict_
.
objectPath
();
}
controlDict_
.
watchIndex
()
=
addWatch
(
f
);
}
}
...
...
@@ -308,19 +327,36 @@ Foam::Time::Time
readLibs_
(
controlDict_
,
"libs"
),
functionObjects_
(
*
this
)
{
// Since could not construct regIOobject with setting:
controlDict_
.
readOpt
()
=
IOobject
::
MUST_READ_IF_MODIFIED
;
setControls
();
// Time objects not registered so do like objectRegistry::checkIn ourselves.
if
(
runTimeModifiable_
)
{
monitorPtr_
.
reset
(
new
fileMonitor
());
monitorPtr_
.
reset
(
new
fileMonitor
(
regIOobject
::
fileModificationChecking
==
inotify
||
regIOobject
::
fileModificationChecking
==
inotifyMaster
)
);
// File might not exist yet.
fileName
f
(
controlDict_
.
filePath
());
if
(
f
!=
fileName
::
null
)
if
(
!
f
.
size
())
{
controlDict_
.
watchIndex
()
=
addWatch
(
f
);
// We don't have this file but would like to re-read it.
// Possibly if in master-only reading mode. Use a non-existing
// file to keep fileMonitor synced.
f
=
controlDict_
.
objectPath
();
}
controlDict_
.
watchIndex
()
=
addWatch
(
f
);
}
}
...
...
src/OpenFOAM/db/Time/TimeIO.C
View file @
b5dddd89
...
...
@@ -211,7 +211,14 @@ void Foam::Time::readModifiedObjects()
// valid filePath).
// Note: requires same ordering in objectRegistries on different
// processors!
monitorPtr_
().
updateStates
(
Pstream
::
parRun
());
monitorPtr_
().
updateStates
(
(
regIOobject
::
fileModificationChecking
==
inotifyMaster
||
regIOobject
::
fileModificationChecking
==
timeStampMaster
),
Pstream
::
parRun
()
);
// Time handling is special since controlDict_ is the one dictionary
// that is not registered to any database.
...
...
src/OpenFOAM/db/regIOobject/regIOobject.C
View file @
b5dddd89
...
...
@@ -36,6 +36,35 @@ int Foam::regIOobject::fileModificationSkew
Foam
::
debug
::
optimisationSwitch
(
"fileModificationSkew"
,
30
)
);
namespace
Foam
{
template
<>
const
char
*
Foam
::
NamedEnum
<
Foam
::
regIOobject
::
fileCheckTypes
,
4
>::
names
[]
=
{
"timeStamp"
,
"timeStampMaster"
,
"inotify"
,
"inotifyMaster"
};
}
const
Foam
::
NamedEnum
<
Foam
::
regIOobject
::
fileCheckTypes
,
4
>
Foam
::
regIOobject
::
fileCheckTypesNames
;
// Default fileCheck type
Foam
::
regIOobject
::
fileCheckTypes
Foam
::
regIOobject
::
fileModificationChecking
(
fileCheckTypesNames
.
read
(
debug
::
optimisationSwitches
().
lookup
(
"fileModificationChecking"
//Foam::regIOobject::timeStamp
)
)
);
// * * * * * * * * * * * * * * * * Constructors * * * * * * * * * * * * * * //
...
...
@@ -149,10 +178,13 @@ bool Foam::regIOobject::checkIn()
}
fileName
f
=
filePath
();
if
(
f
!=
fileName
::
null
)
if
(
!
f
.
size
()
)
{
watchIndex_
=
time
().
addWatch
(
f
);
// We don't have this file but would like to re-read it.
// Possibly if master-only reading mode.
f
=
objectPath
();
}
watchIndex_
=
time
().
addWatch
(
f
);
}
// check-in on defaultRegion is allowed to fail, since subsetted meshes
...
...
src/OpenFOAM/db/regIOobject/regIOobject.H
View file @
b5dddd89
...
...
@@ -41,6 +41,7 @@ SourceFiles
#include
"IOobject.H"
#include
"typeInfo.H"
#include
"OSspecific.H"
#include
"NamedEnum.H"
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
...
...
@@ -57,6 +58,20 @@ class regIOobject
public
IOobject
{
public:
//- Types of communications
enum
fileCheckTypes
{
timeStamp
,
timeStampMaster
,
inotify
,
inotifyMaster
};
static
const
NamedEnum
<
fileCheckTypes
,
4
>
fileCheckTypesNames
;
private:
// Private data
...
...
@@ -95,6 +110,8 @@ public:
static
int
fileModificationSkew
;
static
fileCheckTypes
fileModificationChecking
;
// Constructors
...
...
src/OpenFOAM/db/regIOobject/regIOobjectRead.C
View file @
b5dddd89
...
...
@@ -26,7 +26,7 @@ License
#include
"regIOobject.H"
#include
"IFstream.H"
#include
"Time.H"
//
#include "Pstream
ReduceOps
.H"
#include
"Pstream.H"
// * * * * * * * * * * * * * * * Member Functions * * * * * * * * * * * * * //
...
...
@@ -170,8 +170,50 @@ bool Foam::regIOobject::readData(Istream&)
bool
Foam
::
regIOobject
::
read
()
{
bool
ok
=
readData
(
readStream
(
type
()));
close
();
// Note: cannot do anything in readStream itself since this is used by
// e.g. GeometricField.
bool
masterOnly
=
regIOobject
::
fileModificationChecking
==
timeStampMaster
||
regIOobject
::
fileModificationChecking
==
inotifyMaster
;
bool
ok
;
if
(
Pstream
::
master
()
||
!
masterOnly
)
{
ok
=
readData
(
readStream
(
type
()));
close
();
}
if
(
masterOnly
)
{
// Scatter master data
if
(
Pstream
::
master
())
{
for
(
int
slave
=
Pstream
::
firstSlave
();
slave
<=
Pstream
::
lastSlave
();
slave
++
)
{
OPstream
toSlave
(
Pstream
::
scheduled
,
slave
);
writeData
(
toSlave
);
}
}
else
{
if
(
IFstream
::
debug
)
{
Pout
<<
"regIOobject::read() : "
<<
"reading object "
<<
name
()
<<
" from master processor "
<<
Pstream
::
masterNo
()
<<
endl
;
}
IPstream
fromMaster
(
Pstream
::
scheduled
,
Pstream
::
masterNo
());
ok
=
readData
(
fromMaster
);
}
}
return
ok
;
}
...
...
src/OpenFOAM/global/argList/argList.C
View file @
b5dddd89
...
...
@@ -55,6 +55,12 @@ Foam::argList::initValidTables::initValidTables()
);
argList
::
addBoolOption
(
"parallel"
,
"run in parallel"
);
validParOptions
.
set
(
"parallel"
,
""
);
argList
::
addOption
(
"roots"
,
"(dir1 .. dirn)"
,
"slave root directories for distributed running"
);
validParOptions
.
set
(
"roots"
,
"(dir1 .. dirn)"
);
Pstream
::
addValidParOptions
(
validParOptions
);
}
...
...
@@ -511,6 +517,10 @@ Foam::argList::argList
// Case is a single processor run unless it is running parallel
int
nProcs
=
1
;
// Roots if running distributed
fileNameList
roots
;
// If this actually is a parallel run
if
(
parRunControl_
.
parRun
())
{
...
...
@@ -520,28 +530,42 @@ Foam::argList::argList
// establish rootPath_/globalCase_/case_ for master
getRootCase
();
IFstream
decompDictStream
(
rootPath_
/
globalCase_
/
"system/decomposeParDict"
);
// See if running distributed (different roots for different procs)
label
dictNProcs
=
-
1
;
fileName
source
;
if
(
!
decompDictStream
.
good
(
))
if
(
options_
.
found
(
"roots"
))
{
FatalError
<<
"Cannot read "
<<
decompDictStream
.
name
()
<<
exit
(
FatalError
)
;
IStringStream
str
(
options_
[
"roots"
]);
str
>>
roots
;
dictNProcs
=
roots
.
size
()
+
1
;
source
=
"roots-command-line"
;
}
else
{
source
=
rootPath_
/
globalCase_
/
"system/decomposeParDict"
;
IFstream
decompDictStream
(
source
);
dictionary
decompDict
(
decompDictStream
);
if
(
!
decompDictStream
.
good
())
{
FatalError
<<
"Cannot read "
<<
decompDictStream
.
name
()
<<
exit
(
FatalError
);
}
label
dictNProcs
(
readLabel
dictionary
decompDict
(
decompDictStream
);