Yield CPU for external (e.g. Dask) processes
Damaris server cores sit in the Reactor::RunOne() method they hit MPI_Waitany() and spin the CPU waiting state. This will affect the running of external processes, such as Dask workers, that are spawned using a system call from within Damaris, or externally launched on the same resources as CPU resources will not be shared.
Damaris will try to launch a local Dask worker when scheduler-file
is specified in <pyscript>
tag i.e.
<pyscript name="MyPyAction" file="/damaris-scripts/publish_data.py" language="python"
frequency="1" scheduler-file="/data/dask.json" nthreads="1" keep-workers="no" />
The value of nthreads="1"
is the number of Dask workers to launch locally by the Damaris cores. Set to 0 if the Dask workers need to be launched separately (manually, by a user) and a local worker is not required.
Understanding the best way to set up MPI to reduce the interference of the CPU spin is needed. The methods to modify the behavior of the wait states change with the MPI used:
OpenMPI:
from: scicomp.stackexchange.com cpu-usage-when-a-mpi-rank-waits-during-a-blocking-communication
Aggressive busy wait: --mca mpi_yield_when_idle 0
Degraded busy wait: --mca mpi_yield_when_idle 1
Or, implement a sleep in Damaris: Call MPI_Iprobe() in a loop with a sleep call.
Intel MPI:
from: stackoverflow.com cpu-usage-keeps-100-in-a-do-loop-mpi-bcast-using-intel-mpi
Intel MPI uses environment variables
I_MPI_WAIT_MODE=0
I_MPI_THREAD_YIELD=3
I_MPI_THREAD_SLEEP=10
IBM:
(Possibly out of date)
MP_WAIT_MODE=poll Poll across blocked threads.
MP_WAIT_MODE=yield Suspend blocked threads.
Other details yet to be found...