Introduction
Here we describe the addition of Python support to the Damaris library (from tag v1.7.0) that allows a Python script to be run on each specified iteration of a simulation and have access to the data that are specified within the Damaris XML configuration file. The interface integrates the capabilities of Dask, which is a Python library designed for distributed parallel computing. Dask has a rich infrastructure of data-types and methods that mirror those in popular analytics library Scikit-learn.
Building Damaris with Python support
Integration of Python into the Damaris C++ code base has been facilitated through the use of the Boost::Python library. Boost is a portable and robust set of open-source C++ code and is found in many major Linux distributions within their package/library management systems.
N.B. There is an issue when trying to use the Python functionality at the same time as the Paraview processing functionality (it also uses Python), which seems to derive from the mpi4py library. For details see Damaris Issue 34
Install standard prerequisite libraries:
- xerces-c
- XSD
- cppunit
e.g. On Debain/Ubuntu
sudo apt-get install xsdcxx libxerces-c-dev libcppunit-devOr, from source
# xerces-c
INSTALL_PATH=/home/jbowden/local
wget https://archive.apache.org/dist/xerces/c/3/sources/xerces-c-3.2.2.tar.gz
tar -xf xerces-c-3.2.2.tar.gz
cd xerces-c-3.2.2
./configure --disable-static CC=gcc CXX=g++ CFLAGS=-O3 CXXFLAGS=-O3 --prefix=$INSTALL_PATH
make -j8
make install
# XSD
wget https://www.codesynthesis.com/download/xsd/4.0/xsd-4.0.0+dep.tar.bz2
bzip -d xsd+dep-4.0.0.tar.bz2
tar xf xsd+dep-4.0.0.tar
cd xsd-4.0.0+dep
make CPPFLAGS="-I$INSTALL_PATH/include -std=c++11" LDFLAGS=-L$INSTALL_PATH/lib install install_prefix=$INSTALL_PATH
Install additional prerequisite libraries:
- NumPy See NumPy Install Docs.
- Boost Python and Boost Numpy (incorporated in Boost Python). See Boost Docs.
The Boost NumPy library should be installed if NumPy is available when building boost::python. The Python-dev headers and shared libraries must also be available. - The mpi4py Python module. This module should be installed from source, using pip. See mpi4py.readthedocs.io for installation instructions.
- Dask is recommended but not a hard requirement: See Dask docs for installation instructions.
N.B. Many Linux distributions (e.g. Debian) have pre-packaged Python development libraries and the full set of Boost libraries available through their package manager (e.g. apt-get install ...). Also, see the .python Docker build scripts in the Damaris Gitlab build/docker/bases directory for examples for tested operating system variants.
To enable Python support in Damaris, use the following CMake build system command line.
git clone https://gitlab.inria.fr/Damaris/damaris.git
cd damaris
git fetch --all
# git checkout tags/v1.7.0 - use master as it has current latest updates
cd ..
mkdir -p ./build/damaris
cd build/damaris
cmake ../../damaris -DCMAKE_INSTALL_PREFIX:PATH=$HOME/local \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_COMPILER=$(which mpicxx) \
-DCMAKE_C_COMPILER=$(which mpicc) \
-DENABLE_TESTS=OFF \
-DENABLE_EXAMPLES=ON \
-DBUILD_SHARED_LIBS=ON \
-DGENERATE_MODEL=ON \
-DENABLE_FORTRAN=OFF \
-DENABLE_HDF5=OFF \
-DENABLE_VISIT=OFF \
-DENABLE_CATALYST=OFF \
-DENABLE_PYTHON=ON \
-DENABLE_PYTHONMOD=ON -DPYTHON_MODULE_INSTALL_PATH=$HOME/mypylib
make -j8
make install
# N.B. -DPYTHON_MODULE_INSTALL_PATH could be set to PYTHON_USER_SITE=$(python -m site --user-site)
# Or, make sure you set: export PYTHONPATH=$HOME/mypylib:$PYTHONPATHAn Overview of Capabilities
If a simulation has Damaris functionality, then using Python to access its data is as easy as adding as <pyscript> element to the simulations Damaris XML configuration and specifying which variables can be accessible to the script.
The following is an example of the Damaris XML elements required to enable Python access to a particular variable, named "cube_d":
<variable name="cube_d" type="scalar" layout="cells_whd_wf" mesh="mesh"
centering="nodal" script="MyPyAction" />
<scripts>
<pyscript name="MyPyAction"
file="my_py_script.py" language="python"
frequency="1"
scheduler-file="/home/user/dask_file.json"
nthreads="1"
keep-workers="no"
timeout="4" />
</scripts>
A (for example C++) simulation must the write data to the variable, e.g. initialize and then call Damaris functions:
...
damaris_write("cube_d", (void *) ptr_to_cube_d_data) ;
...
damaris_end_iteration() ;An example Python script my_py_script.py that is read by the simulation Damaris server processes on each iteration and can access the data by name "cube_d" is as follows:
def main(DD):
from dask.distributed import Client, TimeoutError
from damaris4py.server import getservercomm
from damaris4py.dask import damaris_dask
from damaris4py.dask import damaris_stats
try:
damaris_comm = getservercomm()
iteration = damaris_dask.return_iteration(DD)
scheduler_file = damaris_dask.return_scheduler_filename(DD)
if (scheduler_file != ""):
try:
client = Client(scheduler_file=scheduler_file, timeout='2s')
# We now create the Dask dask.array from the Simulation data
x = damaris_dask.return_dask_array(DD,client, 'cube_d', damaris_comm)
# Now, do something with x
except TimeoutError as err:
print('Python ERROR: TimeoutError!: ', err)
except KeyError as err:
print('Python ERROR: KeyError: No damaris data of name: ', err)
else:
print('Python INFO: Scheduler file not found:', scheduler_file)
finally:
pass
if __name__ == '__main__':
main(DamarisData) Running the Examples
There are 3 examples in the examples/python directory that can be useful to understand what is happening. The examples are available in the Damaris install directory under /examples/damaris/python.
Please note: The examples are only installed if CMake was configured with -DENABLE_EXAMPLES=ON. And, if they are installed in a read-only system directory, you will need to copy them to your local directory area. They do not write a lot of data.
Example 1
A very basic example that only uses Python functionality (no Dask). From directory : <INSTALL_DIR>/examples/damaris/python Executable : 3dmesh_py Damaris XML config : 3dmesh_py.xml Python script : 3dmesh_py.py
Usage: 3dmesh_py <3dmesh_py.xml> [-v] [-r] [-s X]
-v <X> X = 0 default, do not print arrays
X = 1 Verbose mode, prints arrays
X = 2 Verbose mode, prints summation of arrays
-r Array values set as rank of process\n");
-s <Y> Y is integer time to sleep in seconds between iterations
-i <I> I is the number of iterations of simulation to run\
mpirun --oversubscribe -np 5 ./3dmesh_py 3dmesh_py.xml -i 3 -v 2 -rExpected result:
For the same iteration, the sum output from C++ printed to screen should match the value computed by the 3dmesh_py.py script
N.B. This example will fail if run over multiple nodes, or with multiple Damaris server cores, as the Python script only has access to NumPy data of the clients that are working with the server core.
To see this in action, change <dedicated cores="1" nodes="0" /> to <dedicated cores="2" nodes="0" /> in file 3dmesh_py.xml and run:
# Note the extra rank requested: -np 6, to account for the extra Damaris server core.
mpirun --oversubscribe -np 6 ./3dmesh_py 3dmesh_py.xml -i 3 -v 2 -rYou will notice 2 outputs from the Python script on each iteration, one output for each Damaris server, showing a sum, and still only a single output of the sum from C++. You will notice that the sum of the 2 Python outputs will be equal to the C++ total.
C++ iteration 0 , Sum () = 98304
...
Python iteration 0 , Sum() = 16384
Python iteration 0 , Sum() = 81920
This shows thae distributed nature of the data, with each Damaris server looking after data from particular Damaris server ranks (2 clients per server in this case).
Example 2
From directory : <INSTALL_DIR>/examples/damaris/python Executable : 3dmesh_py_domains Damaris XML config : 3dmesh_dask.xml Python script : 3dmesh_dask.py
An example of using Python integration with Dask distributed. This version sets deals with the distributed data on the Python side by creating a dask.array, so it can sum of the data in the blocks over multiple Damaris server cores and even over distributed nodes.
To run this example, a Dask scheduler needs to be spun up:
# N.B. best to run this in a separate xterm session as it is quite verbose
dask-scheduler --scheduler-file "$HOME/dask_file.json" &The --scheduler-file argument must match what is specified in the Damaris XML file tag.
To run the simulation:
Assumes 4 Damaris clients and 2 Damaris server cores as per the xml file
i.e. the XML input file (3dmesh_dask.xml) ] contains <dedicated cores="2" nodes="0" /> in the tag.
Usage: 3dmesh_py_domains <3dmesh_dask.xml> [-i I] [-d D] [-r] [-s S];
-i I I is the number of iterations of simulation to run
-r Array values set as rank of process
-d D D is the number of domains to split data into (must divide into WIDTH value in XML file perfectly)
-s S S is integer time to sleep in seconds between iterations
mpirun --oversubscribe -np 6 ./3dmesh_py_domains 3dmesh_dask.xml -i 10 -r -d 4N.B. Set the global mesh size values WIDTH, HEIGHT and DEPTH using the XML input file. Set the name of the Python script to run in the <pyscript ... file="3dmesh_dask.py" ...> tag
The simulation code (via Damaris pyscript class) will create the Dask workers (one per Damaris server core) and have them connect to the Dask scheduler. The simulation code will remove the workers at the end of the execution, unless keep-workers="yes" is specified in 3dmesh_dask.xml tag.
To stop the scheduler:
DASK_SCHEDULER_FILE=$HOME/dask_file.json # must match what you are using above
DASK_SCHED_STR=\'$DASK_SCHEDULER_FILE\'
python3 -c "from dask.distributed import Client; client= Client(scheduler_file=$DASK_SCHED_STR, timeout='2s'); client.shutdown()"Example 3
This example requires an OAR cluster to launch jobs on. There should be equivalent scripts for SLURM and PBS schedulers. From directory : <INSTALL_DIR>/examples/damaris/python/OAR Workflow launcher : stats_launcher.sh Scheduler scripts : stats_launch_scheduler.sh stats_launch_one_job.sh stats_get_results.sh Executable : stats_3dmesh_dask Damaris XML config : stats_3dmesh_dask.xml Python script : stats_3dmesh_dask.py & stats_get_results.py
The example runs multiple versions of the same simulation, with different input values and computes running statistics (averages and variances) of the exposed Damaris field data (cude_d), a rank 3 dimension double precision array. Each simulation runs multiple iterations, each iteration presenting the same value (unique for the simulation instance) to the running statistic arrays.
The stats_launcher.sh workflow launcher script goes through the following steps:
- Get value for number of simulations to run from the command line (if present).
- Shutdown and remove any previous Dask scheduler.
- Launch the Dask scheduler (stats_launch_scheduler.sh) - all jobs subsequent will attach to this scheduler. The name of the scheduler-file must match what is stored in the Damaris XML tag of the simulations (launched in stats_launch_scheduler.sh).
- Poll to wait until Dask scheduler job has stared.
- Launch the simulations (stats_launch_one_job.sh), passing in a different value to add to the dataset ($i) N.B. these jobs implicitly use a Dask scheduler, which is specified in the Damaris XML file tag - it must match the path to DASK_SCHEDULER_FILE
- Wait for OAR jobs to finish.
- Collect the summary statistics (stats_get_results.sh) - stats_get_results.py will print to screen (data is small enough).
- Remove the Dask job scheduler job.
To run the example, log in to Grid5000 and cd to <INSTALL_DIR>/examples/damaris/python/OAR
cd <INSTALL_DIR>/examples/damaris/python/OAR
./stats_launcher.sh 4The results are printed to the job output file stats_finalize_%jobid%.std file and if 4 simulations are launched, each one running 4 iterations and on each iteration adding a value 1,2,3,4, then the mean should be 2.5 and the variance 1.333...
Deep Dive into Damaris Server data structures
Most of the dictionary complexity is wrapped up into a Python module functions that allows a user to ask for a published simulation field by name. An example of use follows:
The raw dictionaries that the Damaris server writes are wrapped in the damaris4py Python module that is installed with Damaris when Python support is requested. For help with these modules, please use pydoc commands:
# Ensure the damaris4py module is on the PYTHONPATH
# This path was specified in the Damaris build by -DPYTHON_MODULE_INSTALL_PATH=<path>
export PYTHONPATH=$HOME/mypylib:$PYTHONPATH
python -m pydoc damaris4py.dask.damaris_dask
python -m pydoc damaris4py.dask.damaris_stats
python -m pydoc damaris4py.server
python -m pydoc damaris4py.client # N.B. a full Damaris client wrapper is planned to be available soon.
Further details of data structures
The Damaris server processes present three dictionaries, damaris_env, dask_env and iteration_data containing various data about the simulation as well as the variable data itself, packaged as Numpy arrays.
DD['damaris_env'].keys() - The global Damaris environment data
(['is_dedicated_node', # True for in-transit processing
'is_dedicated_core', # True for in-situ processing
'servers_per_node', # Number of Damaris server ranks per node
'clients_per_node', # Number of Damaris client ranks per node
'ranks_per_node', # Total number of ranks per node
'cores_per_node', # Total number of ranks per node (yes, the same as above)
'number_of_nodes', # The total number of nodes used in the simulation
'simulation_name', # The name of the simulation (specified in Damaris XML file)
'simulation_magic_number' # Unique number for a simulation run (used in constructing
# name of Dask workers.)
])
DD['dask_env'].keys() - The Dask environment data,
(['dask_scheduler_file', # if an empty string then no Dask scheduler was found
'dask_workers_name', # Each simulation has a uniquely named set of workers
'dask_nworkers' # The total number of dask workers
# (== 'servers_per_node' x 'number_of_nodes')
'dask_threads_per_worker' # Dask workers can have their own threads. Specify as nthreads="1" in Damris XML file
])
The iteration_data dictionary contains the data for a particular iteration, which includes the actual NumPy data for the variables as a list of sub-dictionaries, one for each Damaris variable that has been exposed to the Python interface. i.e. for variables specified with the script="MyAction" as in the example above.
DD['iteration_data'].keys() - A single simulation iteration set of data.
(['iteration', # The iteration number as an integer.
'cube_i', # A Damaris variable dictionary –
# the name matches the variable name used in the Damaris XML file
'...', # Another Damaris variable dictionary
'...' # Another Damaris variable dictionary
])A Damaris variable dictionary has the following structure, as seen for an example of Damaris variable named 'cube_i'
DD['iteration_data']['cube_i'].keys()
(['numpy_data',
'sort_list',
])sort_data is a list, that can be sorted on (possibly required to be transformed to tuple) which when sorted, the list values can be used to reconstruct the whole array using Dask:
DD['iteration_data']['cube_i']['sort_list']A specific example of sort_list:
['S0_I1_<simulation_magic_number>', 'P0_B0', [ 0, 9, 12 ]]- The string 'S0_I1_<simulation_magic_number>' indicates 'S' for server and 'I' for iteration. The magic number is needed as the data can be published to a Dask server from multiple simulations. The magic number value is generated by the Damaris library.
- The string 'P0_B0' indicates the dictionary key of Numpy data (see next description for explanation of 'P' and 'B').
- The list [ 0, 9, 12 ] indicates the offsets into the global array from where the NumPy data is mapped N.B. The size of the NumPy array indicates the block size of the data.
And, finally, the NumPy data is present in blocks, given by keys constructed as described below
DD['iteration_data']['cube_i']['numpy_data'].keys()
(['P0_B0',
'P1_B0'
])Damaris NumPy data keys: 'P' + damaris client number + '_B' + domain number.
- The client number is the source of the data (i.e. it is the Damaris client number/rank that wrote the data to the Damaris server using
damaris_write()ordamaris_write_block()) - The domain number is the result of multiple calls to
damaris_write_block()or 0 if only thedamaris_write()API was used and/or a single block only was written bydamaris_write_block().