Feature/229 extend experiment with testing facility
Description:
This MR provides a new interface for testing facility configuration. Linked to #229 (closed)
It implements:
-
in
Experiment
:
- a setter for testing ratio (
set_test_ratio
) - a setter for testing metrics (
set_test_metric
) - setters for setting flags, that indicates if the testing is occurring before or after the model local updates (
set_flag_test_on_local_updates
,set_flag_on_global_updates
)
-
in
Dataset
(FederatedDataset
):
- a setter for testing ratio (
set_test_ratio
)
Specificities:
- Testing arguments are stored in
training_args
-
TrainRequest
has a new entry in the message:training
, which is a boolean that indicates if training is done or not. In this version, it is used to do model evaluation at the end of several rounds. - when both
FederatedDataset
andExperiment
have differenttest_ratio
,Experiment
will update itstest_ratio
with the value set inFederatedDataset
- By default, testing is done on local update, but this behavior can be changed using
set_flag_test_on_local_updates
andset_flag_on_global_updates
setters.
This new feature still need the following updates to be considered complete:
- accept Fedbiomed metrics in
set_test_metric
- check if
metric_args
are correct inset_test_metric
- accept a default parameter in
set_test_metric
- change the API of
training_routine
: since all arguments are passed in the training_args, andtraining_args
should matchtraining_routine
API, I added some extra argument related to testing in thetraining_routine
API, as a quick and dirty fix. THIS SHOULD BE CHANGED AFTERWARDS
Possible improvements
- allow user to set
test_ratio
as a dictionary mapping : <test_ratio>
Edited by BOUILLARD Yannick