|
|
|
<!-- --- title: Tuning (runtime) -->
|
|
|
|
|
|
|
|
[](/home)
|
|
|
|
[](/building)
|
|
|
|
[][all-repos]
|
|
|
|
|
|
|
|
The OpenFOAM [`etc/controlDict`][etc-controlDict] file contains
|
|
|
|
a list of commonly used user-configurable switches, which are divided
|
|
|
|
into three broad categories:
|
|
|
|
- InfoSwitches :
|
|
|
|
controls output and some system settings
|
|
|
|
- OptimisationSwitches :
|
|
|
|
various *tuning* parameters
|
|
|
|
- DebugSwitches :
|
|
|
|
enable low-level debugging output (mostly for developers or deeper
|
|
|
|
diagnostics when things go wrong)
|
|
|
|
|
|
|
|
In most cases the individual switch entries are *registered* internally
|
|
|
|
by OpenFOAM, which means that they can also be modified by user input.
|
|
|
|
|
|
|
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
|
|
## Changing switch values
|
|
|
|
|
|
|
|
### Making central changes
|
|
|
|
|
|
|
|
If changes are needed on a site-wide basis, the OpenFOAM
|
|
|
|
`etc/controlDict` can be edited directly.
|
|
|
|
This ensures that changes will be used for all jobs and users.
|
|
|
|
However, these changes must be consistent across all compute nodes
|
|
|
|
otherwise the effect is unpredicable. Additionally, changes to the
|
|
|
|
`etc/controlDict` will be lost when changing to a new version.
|
|
|
|
|
|
|
|
|
|
|
|
### Making case-specific changes
|
|
|
|
|
|
|
|
To make case-specific changes, simply add a corresponding sub-dictionary
|
|
|
|
within the case `system/controlDict`. For example,
|
|
|
|
```
|
|
|
|
OptimisationSwitches
|
|
|
|
{
|
|
|
|
// Force dumping (at next timestep) upon signal: 12 = USR2
|
|
|
|
writeNowSignal 12;
|
|
|
|
}
|
|
|
|
```
|
|
|
|
|
|
|
|
### Making job-specific changes
|
|
|
|
|
|
|
|
*(largely v1912 and later)* <br>
|
|
|
|
The different switches can also be adjusted via corresponding
|
|
|
|
command-line options (`-info-switch`, `-opt-switch`, `-debug-switch`).
|
|
|
|
The switch can be specified without a value, in which case
|
|
|
|
a default value is assumed (typically "1", since many switches are
|
|
|
|
integers), or with a specified value.
|
|
|
|
|
|
|
|
Some examples:
|
|
|
|
```
|
|
|
|
$ simpleFoam -info-switch writeOptionalEntries -dry-run
|
|
|
|
|
|
|
|
$ foamDictionary -debug-switch fieldExpr=4 -expand mydict
|
|
|
|
|
|
|
|
$ simpleFoam -opt-switch maxMasterFileBufferSize=1e6
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
## Parallel Tuning
|
|
|
|
|
|
|
|
*(largely v2306 and later)* <br>
|
|
|
|
As part of the [exaFOAM][exafoam] project a variety of communication
|
|
|
|
bottlenecks have been identified and avoided.
|
|
|
|
Some of the different algorithms and settings have been made configurable
|
|
|
|
via OptimisationSwitches (MPI/Parallel settings).
|
|
|
|
|
|
|
|
One of the algorithmic changes is to use
|
|
|
|
NBX (non-blocking exchange) instead of an all-to-all when
|
|
|
|
exchanging buffer sizes, which should improve scaling above 2k ranks
|
|
|
|
and have no disadvantage with lower numbers of ranks.
|
|
|
|
Since these settings are still under testing, they are disable by
|
|
|
|
default but can be selected via OptimisationSwitches.
|
|
|
|
|
|
|
|
|
|
|
|
Some of the OptimisationSwitches relevant for MPI/Parallel:
|
|
|
|
|
|
|
|
- `nbx.min` *NBX min size* (default: 0) <br>
|
|
|
|
The minimum number of processor ranks to use the non-blocking
|
|
|
|
exchange (NBX) algorithm when exchanging sizes.
|
|
|
|
For values less than 1, an all-to-all is used instead.
|
|
|
|
- `nbx.tuning` *NBX tuning parameter* (default: 0) <br>
|
|
|
|
For some systems an initial barrier may be necessary for NBX
|
|
|
|
to avoid race conditions. A value of 1 or greater can currently be
|
|
|
|
used to enable an initial barrier.
|
|
|
|
- `pbufs.tuning` *PstreamBuffers tuning* (default: -1)
|
|
|
|
* **value = -1 (default)** :
|
|
|
|
use all-to-all to exchange buffer sizes,
|
|
|
|
followed by point-to-point communication for the contents.
|
|
|
|
* **value = 0** :
|
|
|
|
use NBX to exchange buffer sizes,
|
|
|
|
followed by point-to-point communication for the contents.
|
|
|
|
* **value = 1** :
|
|
|
|
use NBX exchange for the buffer contents (and sizes) directly.
|
|
|
|
This is not currently recommended until it has been much more
|
|
|
|
thoroughly tested.
|
|
|
|
- `nPollProcInterfaces` *processor patch polling* (default: 0) <br>
|
|
|
|
* **value = 0 (default)** :
|
|
|
|
non-polling. Wait for all exchanges to complete before dispatching.
|
|
|
|
* **value = -1** :
|
|
|
|
wait for any requests to finish and dispatch when possible
|
|
|
|
followed by point-to-point communication for the contents.
|
|
|
|
* **value >= 1** : deprecated (will be removed)
|
|
|
|
|
|
|
|
|
|
|
|
### Comments on parallel tuning
|
|
|
|
|
|
|
|
It is expected that large simulations with many geometry changes
|
|
|
|
will be the primary beneficiary of the NBX algorithm since PstreamBuffers
|
|
|
|
is used there. The use of NBX can be selected separately for PstreamBuffers
|
|
|
|
and for regular size exchanges.
|
|
|
|
|
|
|
|
The full benefits of polling processor interfaces (`nPollProcInterfaces`)
|
|
|
|
remain unclear.
|
|
|
|
|
|
|
|
|
|
|
|
--------
|
|
|
|
|
|
|
|
Copyright (C) 2023 OpenCFD Ltd.
|
|
|
|
|
|
|
|
[exafoam]: https://exafoam.eu/
|
|
|
|
|
|
|
|
[etc-controlDict]: https://develop.openfoam.com/Development/openfoam/-/blob/develop/etc/controlDict
|
|
|
|
[all-repos]: https://develop.openfoam.com/Development/ |
