Reduction involving only a subset of processes
Dear Colleagues,
As discussed with @furmento this morning, following her suggestion (thanks!), I am posting this issue to pursue an email exchange (cf. post-scriptum) related to MPI reductions involving only a subset of processes.
When a reduction involves only a subset of MPI processes, all processes, including those which do not execute any task related to the reduction, create a local copy of the data and then this local copy (which is therefore only initialized on those processes) are implied in the final reduction.
It seems to be due to starpu_mpi_redux_data(comm, handle)
, which induces the creation of the initialization tasks on all processes of the comm
communicator, even if, among them, some are not involved in the reduction.
Note that (although much less critical with respect to the above main issue), according to the execution traces, the initialization tasks seem to be performed twice per process (due to a re-initizalization).
To fix the issue -- without having to handle sub-communicators -- one option was suggested by @thibault that I'm trying to recap (sorry for the possible approximations / incorrect reports):
- for avoiding re-initialization, record the fact that the data was not modified (and thus avoid re-initialization as it is useless);
- for avoiding the send, the root should be aware of which nodes have tackled the data, which (if I understand and re-transcript correctly) could be done in
_starpu_mpi_task_decode_v
by monitoringSTARPU_REDUX
occurrences:- the root -- which owns the data -- would note at each occurrence which node will do the task (and therefore contribute)
- the other nodes note that they will have to send their contribution
Then in
starpu_mpi_redux_data_prio
, the root would only have to go through the list of participants instead of the list of all nodes whereas the other nodes can send their contribution and re-init if they did perform a task.
Hope the transcription if not far from the idea suggested in the email exchange and that this issue submission may help.
Thanks much once again!
With best regards,
@AntJego, @abuttari, @guermouc and Manu
PS: This post follows an email discussion (including motivations for avoiding sub-communicators on the user-side) over the May 7 - 22, 2020 period. I am ccing the recipients @furmento @thibault @aumage