Tests: use robin rather than exec*
Robin is now used to run experiments.
We should think about using it to run Batsim tests. This would imply to:
call robin, either by:
generating many robin input files
calling robin from its CLI from scripts with parameters
write check scripts in specific files (they are currently exec* postcommands)
write wrapper scripts (that call robin then check the result)
It could be the occasion to flatten Batsim tests.
Currently, a test run (a lot of) simulation instances. For each instances, it makes sure the simulation ends correctly [and checks simulation output or not depending on the test].
In a flatten architecture, each simulation instance [+ check] would be a specific test.
Flattened tests can still remain modular, as in CMake a test is just a command to launch.
For example, we could create a script for each current test, and create myriads of tests by calling such scripts multiple times with different parameters.
- Very easy to determine which simulation instance fails.
Currently we have to find it in the execN log (e.g.,
read_csv('instances_info.csv') %>% filter(status=='skipped')).
- Easier to debug, as determining which instance fails and reexecuting it is easier.
- Each instance would require a unique name, so we need some caution when generating them.
- Will generate some CMake noise (but we could use CMake loops to avoid most of it)