- initialization of processor patches is non-blocking, without
  additional buffering

BUG: surface field orientation was being ignored

- when creating processor patches, only negate oriented fields.
  Unoriented fields (eg, temperature) should not be negated.

Sergey Lesnik authored 1 year ago and Mark OLESEN committed 10 months ago

- reading/writing in serial is OK, parallel needs more attention

- The most visible change is the meta-data syntax.
  The attributes are now written in a "dictionary-like" style,
  although actually still streamed as a list of tokens.

    old:
        nonuniform List<scalar> 400 "p/internalField";

    new:
        nonuniform List<scalar> { name "p/internalField"; total 400; };

  Self-documenting and more extensible, at the cost of slightly
  more verbosity.

- reading of the "polyMesh/coherent" dictionary now respects the time
  instance of the polyBoundaryMesh, or can be specified to use an
  alternative timeName

- IFCstream base simplified by reusing existing components for stream
  allocators, reading and broadcast etc.

  The IFCstream reading has been revised to use globalIndex whenever
  possible.  To avoid needless communication, the dictionary/token
  streams are checked for pending tokens. If any are found, the
  globalIndex (eg, the patch size) is calculated once and reused for
  subsequent reading at the similar scope.

  This helps for boundary conditions based on mixed BC, which has four
  fields per patch. For internalField, similarly use the globalIndex
  directly instead of re-creating it again.

    old: label coherentFieldSize();
    new: const globalIndex& coherentInternalOffsets() const;

- use globalIndex semantics when fetching data from ADIOS.

- can now differentiate between scalar and sphericalTensor on reading.
  This was known to have been broken, but was awaiting addition of
  rank information to compoundToken.

- improve robustness of the compoundToken expansion.
  When the tokens are read, they are placed in a "pending" state.
  Later when using ADIOS to populate them, only the pending ones are
  populated. This provisions for situations where some smaller fields
  have been written with a compoundToken but without the ADIOS backend.
  (Currently not possible since these would likely be written in ascii).

Sergey Lesnik authored 1 year ago and Mark OLESEN committed 10 months ago

- IFCstream: extract the type of the field from the compound token and
  not via the template parameter since boundary fields may be of different
  type. The corresponding functions are not templates anymore.

- Field file reading for coherent is done directly in GeometricField.
  If file is not found or the format in the file header is not coherent,
  the standard reading is triggered.

- BUG in OFCstream: write flat fields (label, scalar component type)
  with a flat shape consisting of a single element.

- BUG: CoherentMesh tried to read boundary file although boundary
  information must be extracted from file "coherent". It's tied to
  constant and thus needs a fix in the future.

- BUG: coherentMesh writes every time step. Check write option of the
  points to control the write behavior.

- renumberMesh:
  selects the COHERENT format for writing the mesh and fields.

  Different ways to use it:

  1. renumbering a serial mesh into a corresponding partitioned
     coherent mesh:

      - renumberMesh -decompose -write-coherent
      - renumberMesh -decompose -write-coherent -no-fields
      - renumberMesh -decompose -write-coherent -renumber-method none

  2. a mesh converter preserving the topology/numbering but writing in
     COHERENT format.

      - renumberMesh -write-coherent -renumber-method none -no-fields

- blockMesh:
  the -write-coherent option forces writing as coherent,
  regardless of the controlDict setting

- checkMesh:
  the -coherent option forces reading as coherent,
  regardless of the controlDict setting

- stream each patch entry and modify the output directly.
  This is slightly ugly, but avoids these key issues:

    - cannot write local boundary information since it will have the
      wrong sizes and processor boundaries etc.
    - cannot 'fake' a different set of boundary patches without a
      full mesh to reference.

ENH: respect "requested.partition-offsets" when writing polyMesh as coherent

- normally the mesh partitioning corresponds to the mesh ranks.
  However, when preparing a serial mesh with proper cell ordering
  for a parallel coherent case, the writing will occur in serial.

  For this case, use the polyMesh.data() member [a dictionary]
  to pass the desired partitioning into the writing backend.

ENH: fix search for polyMesh/data.bp/ and polyMesh/coherent

- use the updated Time::findInstance() with new optional parameter to
  avoid re-testing of the result.

ENH: remove old polyMesh/data.bp directory before writing

- avoids an endless appending when the OpenFOAM utility is ostensibly
  in "-overwrite" mode, or when repeatedly calling something like
  blockMesh.

- new wrapping methods (using various pTraits helpers) remove the need
  for the read/write primitives code [will remove in the future]

- use std::string instead of Foam::string for ADIOS identifiers
  since this can be considered low-level infrastructure

- plumb handling of labelPair into lower classes, which mostly
  eliminates the overhead of labelList to convey single or pairs of
  values. There is obviously still a std::vector allocation for
  the final ADIOS dimensions, but at the top-level can reduce overhead
  and conveniently specify shape as first/second.

- change base-level output interface

    OLD:  write(std::unique_ptr<uListProxyBase>)
    NEW:  write(const word&, std::unique_ptr<UListProxyBase>)

  Following the relocation of coherent output into Field instead of
  UList, this provides an interface which is more symmetric with the
  usual writeEntry(key, val) semantics and eliminates the need to
  scan the incoming stream for "List<scalar>" etc.

  This call is done directly from the Field::writeEntry() method
  with the possible compound token type now being originated from
  the UList information.

  The states are now managed by writeKeyword/endEntry
  and beginBlock/endBlock pairs.

- use internal field synchronization points for the field output.

  The GeometricField output is now unwrapped and the boundary patches
  are walked manually. At the begin of each patch (as well as the
  internalField), a sync index is set on the output stream. This sync
  index provides context when creating the individual fieldDataEntry
  items (ie, a correspondence of fieldDataEntry and CoherentMesh
  entities).

- Track fieldDataEntry outside of the collecting dictionary to allow
  directly pruning etc (also avoids recursive gathering).

- Replace the list-based reduction of the uniformity tags/values in
  favour of a direct individual reduction (can be revisited).

COMP: forward declare UListProxyBase, add OFCstream debug flag

WIP: preliminary coherent output for DimensionedField (volMesh only)

Sergey Lesnik authored 1 year ago and Mark OLESEN committed 10 months ago

- Tested by writing volFields with the icoFoam's cavity test case
  (boundary file is still needed in polyMesh)

- ASCII field files have some formatting issues (writing is done with
  dictionary::write)

- OFCstream now inherits from OCharStream; rename foam-extend flavored
  member functions to OF equivalents

- New regIOobject virtual member function "writeToStream" allows to
  construct OFCstream within GeometricField

- The heavy data is given to the OFCstream in the Field class.
  Doing this makes it simple to ensure that every rank will call
  OFCstream::write for every field.

Co-authored-by: Mark Olesen <>

Sergey Lesnik authored 1 year ago and Mark OLESEN committed 10 months ago

- The OFCstream is a special master-only type of buffered writer
  intended for managing lightweight data (dictionary format)
  and heavier backend data (ADIOS) for the COHERENT file format.

  For all ranks it acts a character buffer stream, with periodic
  flushing of accumulated content to create the dictionary content
  as well as various placeholder for the heavier data to be
  redirected to ADIOS.

  On completion, the heavier data will have been written from all
  ranks (via ADIOS) and the lightweight data will have been written on
  the master.

Notes:

  At the moment it seems to still require some intrusive changes
  in other data components.

Co-authored-by: Mark Olesen <>

Sergey Lesnik authored 10 months ago and Mark OLESEN committed 10 months ago

- these low-level components are used as building blocks
  for managing coherent output

Co-authored-by: Mark Olesen <>

ENH: support reading with specified instance

- some functions (eg, Foam::zip) clash with existing OpenFOAM
  functions, even if they were unused

STYLE: remove local Foam::subset template algorithm

- even if not a compilation clash, the Foam::subset has a semantic
  clash with existing OpenFOAM 'subset' functions.
  Unroll outer loop and and use std::copy_if instead.

- reduced overhead and bookkeeping
- non-blocking communication of slices and shared points
- use globalIndex for all internal accounting

- extracts more relevant information during construct or reset
  to allow inexpensive queries such as the number of
  internal/boundary faces and (non-processor) patch sizes etc.

  By consolidating more bookkeeping and functionality can
  greatly reduce the number of scans of input data performed.

- avoid fragile instance counter in favour of construct with index

- support two-stage construction, hide some functions, remove unused
  variables

- align naming more with OpenFOAM conventions.

- include sanity checks on boundary reading, PtrList management

- ensure that "boundary" information does not have more patches
  than known from the coherent format

- polyPatchList instead of List of raw pointers for additional safety.
  Avoids memory leaks, even if the caller doesn't use the data.

Gregor Weiss authored 1 year ago and Mark OLESEN committed 10 months ago

- integration notes (Mark Olesen):

  * left the MUST_READ/READ_IF_PRESENT flags untouched in the
    initializer to reduce noise of the commit (to be addressed)

- also generates a "polyMesh/coherent" dictionary with some boundary
  information etc. Can be useful for later detection of coherent vs
  non-coherent formats etc.

General
-------

- avoid parameter copying, more stringent test on bad input

- support writing primitives with globalIndex reference and labelPair

- obtain underlying component type and number from VectorSpace members
  instead of assuming that these are only 'scalar' and that the data
  buffer is non-null!

- add Reader(), Writer() factory methods for creating streams.
  * Simpler code and few header dependencies.

- style: method name sync_buffer() instead of bufferSync()

- support SYNC mode when reading/writing slice contents.
  Avoids using deferred and then sync on the entire buffer.
  Use get(), get_sync() methods instead of readToContainer helper

Repo
----

- make SliceStreamRepo singleton a std::unique_ptr.
  (avoids leakage on closure).

- add closeInstance() and endInstance() for extra management
  of the singleton

ENH: robuster handling of ADIOS config file

- use "system/adios-config.xml" or "system/adios-config.yaml"
  (was "system/config.xml") to make its intended purpose more transparent.

- although ADIOS does not require a config file, it will fail if the
  config file is specified and missing. Now check the availability of
  the config file(s) and unset if missing.

  NB: checks use the absolute global case-path for additional safety.

ENH: split off read buffer creation and size querying

- add missing totalSize() method

- replace the size() method, which was actually the local size
  with a range() method for querying the local dimensions

  range()     : queries the first component of Start() and Count()
  totalSize() : queries the first component of the Shape()

- add clear(), reset(), clone() methods to InitFromADIOS to simplify
  reuse and multiple queries etc.

ENH: add version (api) and sizes (label,scalar) as attributes

ENH: handle writing of multi-component types directly from SliceStream

- the sliceWritePrimitives helper routines should now be redundant

- cleaner integration of globalIndex:
  * support extract to globalIndex
  * use globalIndex at the components level (for gradual phase-out)

- global end-offsets are directly calculated (using MPI Allgather)

- global start-offsets are calculated with globalIndex::calcOffsets,
  without an intermediate list

- provide constant() DataComponent functor

  * simply returns the constant value with which it was constructed.
    Replace special-purpose start_from_myProcNo, count_two functions
    with equivalent constant() functors

- pass and return strings by reference

ENH: use factory forwarding for InitStrategies to simplify code

- std::unique_ptr ownership passed by move reference

STYLE: place DataComponent functions into OffsetStrategies namespace

- reduces clutter of the Foam namespace
- document their functionality

- replace internal labelPair typedef (std::pair<label,label>, not
  Foam::labelPair) with range_type to avoid possible confusion

- use internal typedefs for OffsetStrategy and DataComponent::base_ptr
  to avoid additional clutter in Foam namespace.

- use class instead of struct with private/protected content

- use OpenFOAM containers and naming conventions.

INT: add transitional methods to Offsets

- Offsets is slated for future removal since it is largely identically
  to what globalIndex provides, but provide compilation path for now

STYLE: simplify/extend sliceMeshHelper

- a namespace with some static functions and/or algorithms for
  handling some of the coherent mesh format details.

 * basic extraction for slice faces / points

 * categorization of upper-connected face neighbours

 * serialize/deserialize owners/faces :
   somewhat similar to CompactListList pack/unpack but
   templated on parameters to support std::vector etc.

  - partition-offsets  (was partitionStarts)
  - owner-offsets      (was ownerStarts)
  - neighbour          (was neighbours)
  - face-offsets       (was faceStarts)
  - face-points        (was faces)

STYLE: adjust class naming to reflect (current/future) purpose

- coherentMeshPolyMapper (old: FragmentPermutation)
- coherentProcessorPatch (old: ProcessorPatch)
- polyMeshCoherentMapper (old: SlicePermutation)

STYLE: use contains() for slice maps

- combine DataComponentFree into DataComponent

- make sliceMap header-only (will be removed)

GIT: remove remnant OffsetDecorator header

- this will presumably change again in the future

Sergey Lesnik authored 1 year ago and Mark OLESEN committed 10 months ago

- the is_coherentIOobject trait is now used directly in typeHeaderOk()
  with SFINAE.

    IOobject header(...);
    header.typeHeaderOk<volScalarField>(true);

  This change allows most of the coherent read logic to be placed
  into IOobject.

- check the system/controlDict for the "writeFormat" value to see if
  coherent output is requested

  With coherent writeFormat detected:

  - disable checks for processor directories.

    Rationale:
        They will either not be needed (FUTURE), or will possibly be
        created on the fly.

  - automatically switch from "uncollated" to "collated".

    Rationale:
        Consider as a degenerate combination since it will potentially
        generate far too many files. The collated format provides a
        much better dumping place for any fields and fields that are
        not yet supported in coherent format.

  - automatically create processorsNNN/ directory for collated format.

    Rationale:
        We've disabled processor directory checking earlier, but
        may potentially still need the directories when checking if
        other files can be read (need to verify). Will nonetheless
        improve robustness with minimal overhead.

- additional streamFormat::UNKNOWN_FORMAT enumeration
  as return code for failure or detect (for example)

- provide is_coherentIOobject type trait

- exprValueFieldTag is an extended version of exprValue,
  with additional Field/List uniformity handling

- the exprValueFieldTag reduce() method provides a more efficient
  method than using a regular combine operator.
  Since fields are often to be non-uniform, can use a bitwise reduce
  most of the time.

ENH: output of exprValue (scalar type) now includes '.'

- avoids scalar/label ambiguity for values like (100.0), which would
  otherwise be written as '100' and thus inadvertently interpreted as
  a label value upon re-reading.

- evaluate()
  the communication type is exposed as a parameter to allow for more
  tuning, but default parameter remains defaultCommsType so there is
  no change in behaviour for existing code

- evaluate_if()
  supports a general selection predicate

- evaluateSelected()
  now does initEvaluate() for all patches, waits and then calls
  evaluate(). This avoids potential deadlocks when multiple patches
  are inter-communicating.

ENH: align DimensionedField reading with GeometricField treatment

- use localIOdictionary to obtain the dictionary contents

STYLE: various

- change GeometricField writeData() as primary output method
  (not operator<<) for better clarity of purpose

- boundary entries with writeEntry(const word&, ...) instead of
  writeEntry(const keyType&, ...) to match with most other
  writeEntry() signatures. Also, this content will not be used
  to supply regex matched sub-dictionaries.

- use unique_ptr for GeometricField demand-driven data

- avoid spurious doxygen generation for GeometricBoundaryField