rewrite Experiment() results query interface
Rewrite results query interface for Experiment()
results based on #170 :
- may include a new class for handling experiment results
- hide the underlying data structure and storage to provide a simple/portable access
- eg:
for round in e.rounds():
for node in e.nodes():
e.display_results(round, node, optional_filters)
or
for round in e.rounds():
for node in e.nodes():
e.results().display(round, node, optional_filters)
Misc:
- data structure for returned value of
exp.check_model_status()
- currently returns aResponses
which is not a good idea
From issue #170:
Experiment results should include all what is needed to analyze an experiment after it completed.
Currently we access an experiment's results by browsing these data structures:
-
exp.aggregated_params
for researcher-computed results -
exp.training_replies
for node-returned results -
exp._rounds
,exp._model_path
for experiment definition (not really part of the experiment results, but probably needed to use the experiment results)
What is lacking:
- no clear definition of what the results of an experiment include
- no class design
Proposed improvements
-
define what scenarios we want to support for an experiment that impact content of results - eg: change
Strategy
, model,Aggregator
during experiment ? (and keep record of the one used for each round)
- eg: change
-
define precisely what is included in the results of an experiment - add from previous point
- misc: add initial value of aggregated parameters ? add experiment call prototype ?
-
define access conditions for an experiment result - eg: access by an
Experiment.results()
in a notebook - eg: later reload of results after
Experiment
is finished and deleted, maybe without instantiating an experiment. Eg.
- eg: access by an
res = Results(path_to_results_of this experiment)
res.results()
-
define methods/interfaces to Experiment
for querying results -
(optional) macro-choice for implementation. Eg how to store the results: - option 1:
Experiment
queries aStrategy.results()
- option 2:
Experiment
queries aResults.strategy_results()
- etc.
- option 1:
Edited by CANSIZ Sergen