Implement loss regularizers & revise Optimizer API.
The main objective of this MR is to add a new type of optimizer plug-ins to declearn, that implement loss-regularization terms through gradients correction (i.e. computing the gradients of the actual regularization term and adding them to the base gradients output by the framework-specific Model code).
One of the outcomes of this new API will be to add support for FedProx, which consists in adding a proximal term to the loss, regularizing it by the (scaled) squared difference between local and global weights (recomputed at each step).
As a side objective, the MR intends to partially revise the OptiModule and, consequently, Optimizer configuration / (de)serialization API to make it easier for end-users to write down and/or edit full optimizer configurations manually.
Tasks:
-
Implement a RegularizerAPI and add it toOptimizer. -
Revise OptiModule(de)serialization format. -
Document Regularizerand updateOptiModuledocumentation (notably in README). -
Deploy and document the revised Optimizerconfiguration syntax (in examples and README).