API: Broadcast root after dynamic start
This is a problem that Dominik and me discussed about recently, I just wanted to write it down so that we don't forget about it.
After a new process set is added (via ADD,GROW,...), a new process set/communicator is usually created that spans both the existing processes and the new one. Then, usually, the application wants to transfer the application state to the newly created processes. This usually happens via a broadcast.
However, it is not easy to determine the root-argument of the broadcast. It is necessary that it is not a process that was just started.
In my preliminary LibPFASST implementation that uses libmpidynres, I solved this issue via a MPI_Allreduce
: https://github.com/boi4/LibPFASST/blob/87b46b0b2b8037e43fcd3630b617119ce8d48c07/src/pf_dynres.f90#L282 (rank 0 of the previous "main" communicator has sync_root set to true).
This is a rather complicated solution and is quite likely to be necessarily done (or something similar) by most applications that use the API in the current state.
Maybe we can avoid this issue by adding some guarantees about the ranks (for example, rank 0 is guaranteed to be the rank 0 from the previous process set).