Skip to content

Feature distributed DIC/DILU preconditioners

Mattijs Janssens requested to merge feature-distributedDIC into develop

Summary

The current DIC/DILU preconditioners operate on the local data only. This merge implements parallel versions of these using a processor-colouring strategy where data from lower coloured processors is included in the forward sweep and (data from) higher coloured processors in the backwards sweep. This generally leads to lower number of PCG sweeps so less global reductions at the cost of more halo swaps.

Details of new models (If applicable)

Implements two new preconditioners: distributedDIC and distributedDILU. Both can be used in a top-level PCG or PBiCGStab solver or for the coarsest-level in GAMG:

p
{
    solver          GAMG;
    // Explicit specify solver for coarse-level correction to override
    // preconditioner
    coarsestLevelCorr
    {
        solver          PCG;
        preconditioner  distributedDIC;
    }
}

Risks

  • for extreme decompositions (e.g. using random) might increase simulation time
  • increased memory usage - allocates buffers for all remote data
  • construction of preconditioners requires halo-swaps (for inverse of diagonal) so might not be cost-effective for low numbers of PCG sweeps
  • communication pattern now has cross-overs - order of receiving is not the same as order of sending. Solved by using different message tags.
  • effectiveness depends on cost of reduction v.s. halo-swaps

Merge request reports