Mentions légales du service

Skip to content

Feature/229 extend experiment with testing facility

Description:

This MR provides a new interface for testing facility configuration. Linked to #229 (closed)

It implements:

  1. in Experiment:
  • a setter for testing ratio (set_test_ratio)
  • a setter for testing metrics (set_test_metric)
  • setters for setting flags, that indicates if the testing is occurring before or after the model local updates (set_flag_test_on_local_updates, set_flag_on_global_updates)
  1. in Dataset(FederatedDataset):
  • a setter for testing ratio (set_test_ratio)

Specificities:

  • Testing arguments are stored in training_args
  • TrainRequest has a new entry in the message: training, which is a boolean that indicates if training is done or not. In this version, it is used to do model evaluation at the end of several rounds.
  • when both FederatedDataset and Experiment have different test_ratio, Experiment will update its test_ratio with the value set in FederatedDataset
  • By default, testing is done on local update, but this behavior can be changed using set_flag_test_on_local_updates and set_flag_on_global_updates setters.

This new feature still need the following updates to be considered complete:

  • accept Fedbiomed metrics in set_test_metric
  • check if metric_args are correct in set_test_metric
  • accept a default parameter in set_test_metric
  • change the API of training_routine: since all arguments are passed in the training_args, and training_args should match training_routine API, I added some extra argument related to testing in the training_routine API, as a quick and dirty fix. THIS SHOULD BE CHANGED AFTERWARDS

Possible improvements

  • allow user to set test_ratio as a dictionary mapping : <test_ratio>
Edited by BOUILLARD Yannick

Merge request reports