Draft: reduced overhead and non-blocking transfers for collated and masterOnly file handling

The aim of these changes is to avoid duplicate copies of character data and improved communication throughput.

The buffering in the output stream are now based on OCharStream instead of OStringStream. This allows full recovery of the streamed characters without additional copies. The character data are "yielded" from the streaming buffer to pass on to the backend writers without an intermediate copy into string and copy back out of a string. The full buffer, including unused portions, is transferred to avoid triggering any alloc/free at that point. The char data can then be directly communicated (non-blocking) to the output.

On the receiving end, the size of the character content can be established directly from an MPI_Probe prior to setting up the MPI_Recv/MPI_Irecv. This avoids both the memory overhead of PstreamBuffers (the previous implementation) as well as needing to coordinate between all ranks (the PstreamBuffers has a synchronization point when establishing the buffer sizes as part of the PEX algorithm).

When using a master-only writing (non-collated), now use polling dispatch to write file content when it becomes available.

changed milestone to %v2406

added enhancement label

requested review from @andy

assigned to @Mattijs

@Prashant - would like to get some tests for this too. Will work out the details off-line.

icoFoam -parallel -fileHandler collated -opt-switch maxThreadFileBufferSize=-1 -opt-switch maxMasterFileBufferSize=200e9

Prints

collated [unthreaded] (maxThreadFileBufferSize = 0).
         With non-blocking transfer, buffer-size = 2e+11

Shouldn't -1 switch on threading upon writing?

On lid-driven cavity with writeFormat binary:

decomposePar -fileHandler collated
mpirun -np 9 icoFoam -parallel -fileHandler collated
mpirun -np 9 redistributePar -parallel -reconstruct -latestTime

fails with

Wrong token type - expected word, found on line 0: label 4

on all processors

Fixed by same issue as #3105 (closed)

gl3105.diff

added 22 commits

• cd795763...46e1b00c - 19 commits from branch develop
• 79732fe5 - ENH: reduce overhead of masterOFstream
• 38a2be91 - ENH: reduce overhead of decomposedBlockData IO
• 05f6934f - ENH: reduce overhead and clearer data ownership for OFstreamCollator

Compare with previous version

marked this merge request as ready

Updated onto the latest develop. Taking off "draft" and see what people find.

resolved all threads

added 81 commits

• 05f6934f...1406f9ec - 77 commits from branch develop
• 4c4e6156 - ENH: improve OFstream append behaviour (#3160 (closed))
• 577509f4 - ENH: reduce overhead of masterOFstream
• d0cb4a5f - ENH: reduce overhead of decomposedBlockData IO
• bd6b8e5c - ENH: reduce overhead and clearer data ownership for OFstreamCollator

Compare with previous version

changed milestone to %v2412

added 84 commits

• bd6b8e5c...dce009ce - 81 commits from branch develop
• f1ece719 - ENH: reduce overhead of masterOFstream
• 90d8f452 - ENH: reduce overhead of decomposedBlockData IO
• f94fa700 - ENH: reduce overhead and clearer data ownership for OFstreamCollator

Compare with previous version

rebased onto current develop. Would be good to get tested and integrated this year.

marked this merge request as draft

unassigned @Mattijs

removed review request for @andy