design Experiment() results query interface
Experiment results should include all what is needed to analyze an experiment after it completed.
Currently we access an experiment's results by browsing these data structures:
-
exp.aggregated_params
for researcher-computed results -
exp.training_replies
for node-returned results -
exp._rounds
,exp._model_path
for experiment definition (not really part of the experiment results, but probably needed to use the experiment results)
What is lacking:
- no clear definition of what the results of an experiment include
- no class design
Proposed improvements
-
define what scenarios we want to support for an experiment that impact content of results - eg: change
Strategy
, model,Aggregator
during experiment ? (and keep record of the one used for each round)
- eg: change
-
define precisely what is included in the results of an experiment - add from previous point
- misc: add initial value of aggregated parameters ? add experiment call prototype ?
-
define access conditions for an experiment result - eg: access by an
Experiment.results()
in a notebook - eg: later reload of results after
Experiment
is finished and deleted, maybe without instantiating an experiment. Eg.
- eg: access by an
res = Results(path_to_results_of this experiment)
res.results()
-
define methods/interfaces to Experiment
for querying results -
(optional) macro-choice for implementation. Eg how to store the results: - option 1:
Experiment
queries aStrategy.results()
- option 2:
Experiment
queries aResults.strategy_results()
- etc.
- option 1:
Edited by VESIN Marc