Reduction involving only a subset of processes
As discussed with @furmento this morning, following her suggestion (thanks!), I am posting this issue to pursue an email exchange (cf. post-scriptum) related to MPI reductions involving only a subset of processes.
When a reduction involves only a subset of MPI processes, all processes, including those which do not execute any task related to the reduction, create a local copy of the data and then this local copy (which is therefore only initialized on those processes) are implied in the final reduction.
It seems to be due to
starpu_mpi_redux_data(comm, handle), which induces the creation of the initialization tasks on all processes of the
comm communicator, even if, among them, some are not involved in the reduction.
Note that (although much less critical with respect to the above main issue), according to the execution traces, the initialization tasks seem to be performed twice per process (due to a re-initizalization).
To fix the issue -- without having to handle sub-communicators -- one option was suggested by @thibault that I'm trying to recap (sorry for the possible approximations / incorrect reports):
- for avoiding re-initialization, record the fact that the data was not modified (and thus avoid re-initialization as it is useless);
- for avoiding the send, the root should be aware of which nodes have tackled the data, which (if I understand and re-transcript correctly) could be done in
- the root -- which owns the data -- would note at each occurrence which node will do the task (and therefore contribute)
- the other nodes note that they will have to send their contribution
starpu_mpi_redux_data_prio, the root would only have to go through the list of participants instead of the list of all nodes whereas the other nodes can send their contribution and re-init if they did perform a task.
Hope the transcription if not far from the idea suggested in the email exchange and that this issue submission may help.
Thanks much once again!
With best regards,
PS: This post follows an email discussion (including motivations for avoiding sub-communicators on the user-side) over the May 7 - 22, 2020 period. I am ccing the recipients @furmento @thibault @aumage