parallel

We are happy to incorporate content from volunteers!!

Coding Patterns - parallel communication

This is a very broad ranging topic, but here are some general ideas...

• Coding Patterns - parallel communication
  • General Considerations
  • One-sided Gather
  • Broadcast
  • Plain Reductions
  • Other Reductions

General Considerations

• When using point-to-point communication, optimize for contiguous data and use the UIPstream::read and UOPstream::write methods.

  UIPstream::read
  (
      Pstream::commsTypes::scheduled,
      fromProci,
      fld1.data_bytes(),
      fld1.size_bytes(),
      tag,
      comm
  );

  and

  UOPstream::write
  (
      Pstream::commsTypes::scheduled,
      toProci,
      fld2.cdata_bytes(),
      fld2.size_bytes(),
      tag,
      comm
  );

  This will use MPI_Recv and MPI_Send directly, without the expense of a MPI_Probe call.

One-sided Gather

• In cases where flattened information is required on the master process only, using globalIndex will frequently provide the most efficient method. Quite frequently, the overall size/offset information is only needed on the master and not on the sub-processes:

  globalIndex globIdx(send.size(), globalIndex::gatherOnly{});

  Using the globalIndex::gatherOnly{} tag will trigger an MPI gather without the expense of needless broadcast to all nodes.

  In many cases, it can be convenient to use the static gather() method:

  const scalarField mergedWeights(globalIndex::gather(weights));

  but can also reuse existing size/offset information:

  const scalarField mergedWeights(globalFaces().gather(weights));

Broadcast

Sending data from the master rank to sub-ranks

Prior to OpenFOAM-v2206, the Foam::scatter() function was used to send data from the master rank to all sub-ranks. This function (which used manual tree communication) is superseded by Pstream::broadcast() which is based instead on MPI Bcast.

The OpenFOAM Pstream::broadcast() function corresponds an MPI Bcast, but with serialize/deserialize wrappers (where needed). For contiguous data types, it maps exactly to an MPI_Bcast. It is an no-op in non-parallel code.

When data are serialized/deserialized, it can be useful to bundle several smaller data items into a single broadcast with the Pstream::broadcasts() function.

Plain Reductions

The OpenFOAM Foam::reduce() function corresponds to an MPI Allreduce, but with serialize/deserialize wrappers (where needed). For primitive data types (eg, bool, int, double, ...) it corresponds exactly to an MPI_Allreduce. It is an no-op in non-parallel code.

The Foam::returnReduce function is used in places where a reduced value is required (eg, for an if statement), or where the value to be reduced in read-only. For example,

    Info<< "Total " << returnReduce(list.size(), sumOp<label>()) << nl;

The reduction of boolean values is treated as a special case, since these be mapped directly to MPI intrinsic calls. Additionally, for the return-reduce case, it can be extremely handy to use the implicit bool conversion. For example,

    // Has values everywhere?
    if (returnReduceAnd(list.size())) ...

    // Any values anywhere?
    while (returnReduceOr(list.size()) && iter < 10) ...

The plain (non-returning versions) can be used as expected:

Pstream::reduceAnd(haveLib);
Pstream::reduceOr(hasError);

Other Reductions

In addition to regular reductions, it is also possible to perform reduction with different types of containers and also with a combine operation. The following are some of the common operations:

Copyright (C) 2022 OpenCFD Ltd.