Implement Metric API.
This MR addresses issue #6 (closed) and implements an API to iteratively and federatively compute evaluation metrics.
The implementation went through iterative steps, that may be reviewed incrementally:
-
Implement a MetricAPI for the iterative and federative computation of evaluation metrics. -
Enable computing metrics using that API as part of TrainingManager.evaluation_round. -
Enable FederatedServerto instruct its clients to compute arbitrary metrics. -
Enable FederatedClientto prevent metrics from being shared due to privacy concerns. -
Update the README and Heart UCI example to document the new features. -
Enhance metrics' reporting and saving by revising Checkpointer(separate MR).
Summary of changes (current state):
- Add a
declearn.metricssubmodule. - Implement a
MetricAPI, with an abstract base class and associated test suite. - Implement standard classification and regression metrics.
BinaryAccuracyPrecisionRecallMulticlassAccuracyPrecisionRecallBinaryRocAucMeanAbsoluteErrorMeanSquaredError
- Implement
MetricSetwrapper forMetricinstances. - Update Model API for better articulation with the Metric API.
- Add
Model.compute_batch_predictionsfor inference. - Add
Model.loss_functionto finalize sample-wise loss computation. - Abstract
Model.compute_lossinto the base class and deprecate it.
- Add
- Deploy the Metrics API to
TrainingManager.- Add
metricsattribute and init parameter toTrainingManager. - Force the wrapping of the model's loss into an ad-hoc
Metric. - Revise the
evaluation_roundbackend to use the wrappedMetricSet. - Report the loss and metric states as part of the returned
EvaluationRequest.
- Add
- Enable
FederatedServerto instruct its clients to compute metrics.- Add
metricsattribute and init parameter toFederatedServer. - Have
FederatedClientreceive the derivedMetricSetconfig during the initialization phase. - Have
FederatedServercollect and aggregated client-emitted metric states.
- Add
- Enable
FederatedClientto prevent sharing evaluation metrics (apart from the scalar loss).- Add
share_metricsboolean switch toFederatedClient(default=True). - Ensure
FederatedServeris resilient to clients' metrics-sharing denial.
- Add
- Update the README file and Heart UCI example to document and showcase metrics use.
Edited by ANDREY Paul