Mentions légales du service

Skip to content

Implement Metric API.

ANDREY Paul requested to merge metrics into main

This MR addresses issue #6 (closed) and implements an API to iteratively and federatively compute evaluation metrics.

The implementation went through iterative steps, that may be reviewed incrementally:

  • Implement a Metric API for the iterative and federative computation of evaluation metrics.
  • Enable computing metrics using that API as part of TrainingManager.evaluation_round.
  • Enable FederatedServer to instruct its clients to compute arbitrary metrics.
  • Enable FederatedClient to prevent metrics from being shared due to privacy concerns.
  • Update the README and Heart UCI example to document the new features.
  • Enhance metrics' reporting and saving by revising Checkpointer (separate MR).

Summary of changes (current state):

  • Add a declearn.metrics submodule.
  • Implement a Metric API, with an abstract base class and associated test suite.
  • Implement standard classification and regression metrics.
    • BinaryAccuracyPrecisionRecall
    • MulticlassAccuracyPrecisionRecall
    • BinaryRocAuc
    • MeanAbsoluteError
    • MeanSquaredError
  • Implement MetricSet wrapper for Metric instances.
  • Update Model API for better articulation with the Metric API.
    • Add Model.compute_batch_predictions for inference.
    • Add Model.loss_function to finalize sample-wise loss computation.
    • Abstract Model.compute_loss into the base class and deprecate it.
  • Deploy the Metrics API to TrainingManager.
    • Add metrics attribute and init parameter to TrainingManager.
    • Force the wrapping of the model's loss into an ad-hoc Metric.
    • Revise the evaluation_round backend to use the wrapped MetricSet.
    • Report the loss and metric states as part of the returned EvaluationRequest.
  • Enable FederatedServer to instruct its clients to compute metrics.
    • Add metrics attribute and init parameter to FederatedServer.
    • Have FederatedClient receive the derived MetricSet config during the initialization phase.
    • Have FederatedServer collect and aggregated client-emitted metric states.
  • Enable FederatedClient to prevent sharing evaluation metrics (apart from the scalar loss).
    • Add share_metrics boolean switch to FederatedClient (default=True).
    • Ensure FederatedServer is resilient to clients' metrics-sharing denial.
  • Update the README file and Heart UCI example to document and showcase metrics use.
Edited by ANDREY Paul

Merge request reports