Implement Metric API.
This MR addresses issue #6 (closed) and implements an API to iteratively and federatively compute evaluation metrics.
The implementation went through iterative steps, that may be reviewed incrementally:
-
Implement a Metric
API for the iterative and federative computation of evaluation metrics. -
Enable computing metrics using that API as part of TrainingManager.evaluation_round
. -
Enable FederatedServer
to instruct its clients to compute arbitrary metrics. -
Enable FederatedClient
to prevent metrics from being shared due to privacy concerns. -
Update the README and Heart UCI example to document the new features. -
Enhance metrics' reporting and saving by revising Checkpointer
(separate MR).
Summary of changes (current state):
- Add a
declearn.metrics
submodule. - Implement a
Metric
API, with an abstract base class and associated test suite. - Implement standard classification and regression metrics.
BinaryAccuracyPrecisionRecall
MulticlassAccuracyPrecisionRecall
BinaryRocAuc
MeanAbsoluteError
MeanSquaredError
- Implement
MetricSet
wrapper forMetric
instances. - Update Model API for better articulation with the Metric API.
- Add
Model.compute_batch_predictions
for inference. - Add
Model.loss_function
to finalize sample-wise loss computation. - Abstract
Model.compute_loss
into the base class and deprecate it.
- Add
- Deploy the Metrics API to
TrainingManager
.- Add
metrics
attribute and init parameter toTrainingManager
. - Force the wrapping of the model's loss into an ad-hoc
Metric
. - Revise the
evaluation_round
backend to use the wrappedMetricSet
. - Report the loss and metric states as part of the returned
EvaluationRequest
.
- Add
- Enable
FederatedServer
to instruct its clients to compute metrics.- Add
metrics
attribute and init parameter toFederatedServer
. - Have
FederatedClient
receive the derivedMetricSet
config during the initialization phase. - Have
FederatedServer
collect and aggregated client-emitted metric states.
- Add
- Enable
FederatedClient
to prevent sharing evaluation metrics (apart from the scalar loss).- Add
share_metrics
boolean switch toFederatedClient
(default=True). - Ensure
FederatedServer
is resilient to clients' metrics-sharing denial.
- Add
- Update the README file and Heart UCI example to document and showcase metrics use.