Benchmarks: ideas
It could be interesting to let user run benchmarks in order to compare two different scripts.
In such a benchmark mode, user could choose:
- a benchmark dataset
- for each script, the number of run to perform (1 by default, meaning the script do not belong to the benchmark)
- eventual alternates implementations. Each script combination can be run independantly, allowing comparison of two or more methods
- whether or not computing the visualization (allowing bigger datasets, without wait for an unreadable graph to be generated)
Simple algorithm:
- assign to each script a number of run
- for script in ordered activated scripts
- if its number of run is 1
- run it
- if its number of run is >1
- run it number of run times
- show the best time, the mean and stddev to the script
- if its number of run is 1
This allow to measure multiple times the same script with the same input, leading to a much more precise runtime measure.
Edited by BOURNEUF Lucas