Implement loss regularizers & revise Optimizer API.

The main objective of this MR is to add a new type of optimizer plug-ins to declearn, that implement loss-regularization terms through gradients correction (i.e. computing the gradients of the actual regularization term and adding them to the base gradients output by the framework-specific Model code).

One of the outcomes of this new API will be to add support for FedProx, which consists in adding a proximal term to the loss, regularizing it by the (scaled) squared difference between local and global weights (recomputed at each step).

As a side objective, the MR intends to partially revise the OptiModule and, consequently, Optimizer configuration / (de)serialization API to make it easier for end-users to write down and/or edit full optimizer configurations manually.

Tasks:

• Implement a Regularizer API and add it to Optimizer.
• Revise OptiModule (de)serialization format.
• Document Regularizer and update OptiModule documentation (notably in README).
• Deploy and document the revised Optimizer configuration syntax (in examples and README).

added feature-request label

assigned to @paandrey

added 1 commit

• 688e2ed4 - Use framework-specific Vector type as `Model` weights.

Compare with previous version

added 2 commits

• f0db5f2d - Fix `Regularizer` API: move name class attribute definition.
• a21f6305 - Fix `FedAvg` due to `Optimizer` API changes.

Compare with previous version

added 6 commits

• 0a1d967f - Revise `create_types_registry`.
• bde421d1 - Rename private `optimizer.modules._base` to `_api` for readability.
• 53dd9eaf - Implement `Regularizer` optimizer plug-ins API.
• e1479287 - Implement FedProx, Lasso and Ridge loss regularizers.
• fd1c012d - Added loss regularizers to `Optimizer`.
• 61d730e4 - Use framework-specific Vector type as `Model` weights.

Compare with previous version

added 2 commits

• 53b27196 - Use framework-specific Vector type as `Model` weights.
• 04bf1448 - Fix `InitRequest.to_string` with tensorflow 2.11.

Compare with previous version

added 2 commits

• 37e505b9 - Improve `Regularizer` docstrings.
• 2e3c0bbc - Revise `OptiModule` API.

Compare with previous version

added 1 commit

• 36c3b760 - Revise `OptiModule` API.

Compare with previous version

added 2 commits

• 8c799416 - Harmonize "Parameters" section naming in docstrings.
• 83e2c96b - Modularize `regularizers` and `modules` input types for `Optimizer`.

Compare with previous version

added 2 commits

• 3866b705 - Apply minor docstring and type-hint corrections.
• 72cbcbac - Refactor `OptiModule` unit tests and implement `Regularizer` ones.

Compare with previous version

added 1 commit

• f00fa033 - Move `NumpyVector` from `model.api` to `model.sklearn` submodule.

Compare with previous version

added 5 commits

• 0887fdd3 - Revise `OptiModule` API.
• 2ed8e450 - Modularize `regularizers` and `modules` input types for `Optimizer`.
• c88c122b - Apply minor docstring and type-hint corrections.
• a63c27b4 - Refactor `OptiModule` unit tests and implement `Regularizer` ones.
• 2fa68022 - Move `NumpyVector` from `model.api` to `model.sklearn` submodule.

Compare with previous version

added 3 commits

• e2da296b - Document `Regularizer` API in the README file.
• b820ce8d - Update Heart-UCI example using new `Optimizer` config syntax.
• 216ba36e - Increment version to "2.0.0.beta2".

Compare with previous version

marked the checklist item Revise OptiModule (de)serialization format. as completed

marked the checklist item Document Regularizer and update OptiModule documentation (notably in README). as completed

marked the checklist item Deploy and document the revised Optimizer configuration syntax (in examples and README). as completed

@nbigaud When you have a moment, could you please review this MR?

Summary of changes:

• New Regularizer API

  • Regularizer is a new type of Optimizer plug-in to implement loss regularization terms. Technically, it computes the derivative of the regularization term based on the current model weights (and/or loss-derived gradients).
  • Automated type-registration of subclasses using __init_subclass__ and relying on the name class attribute.
  • Implemented a from_specs generic constructor to instantiate any registered regularizer from its name and config dict.

• Revised OptiModule API

  • Deprecated the use of OptiModule.deserialize and OptiModule.serialize.
  • Implemented __init_subclass__ tricks and from_specs method similarly to Regularizer.
  • Added aux_name optional (default=None) class attribute to label auxiliary variables when required (e.g. for Scaffold).

• Revised Optimizer API

  • Added regularizers parameter and attribute to use a pipeline of loss regularizers (similarly to modules).
  • Changed to_config and from_config to use a more human-readable format.
  • Modularized the type of input regularizers and modules so that users may either pass instantiated objects, class name keywords or (name, config) tuples.

• "Out of Scope" API changes:

  • Modified declearn.utils.create_types_registry to enable its use as a class decorator.
  • Modified Model.get_weights and Model.set_weights: use subclass-specific Vector rather than NumpyVector.
  • Moved NumpyVector from declearn.model.api to declearn.model.sklearn submodule as it is no longer central.

• Maintenance operations:

  • Revised and added unit tests for optimizer plug-in classes.
  • Updated the README and Heart-UCI example code based on API changes.
  • Pushed version number of 2.0.0.beta2 due to API-breaking changes.

marked this merge request as ready

requested review from @nbigaud

added 1 commit

• a625c91b - Update .gitlab-ci.yml file

Compare with previous version