diff --git a/README.md b/README.md
index 2ae56b166039ba614678650d88432dce5cf2e86b..400c3928d7818d35630add55dbe5b1aed8a0c310 100644
--- a/README.md
+++ b/README.md
@@ -4,17 +4,14 @@
- [Setup](#setup)
- [Quickstart](#quickstart)
- [Usage of the Python API](#usage-of-the-python-api)
- - [Overview of the Federated Learning process](#overview-of-the-federated-learning-process)
- - [Overview of the declearn API](#overview-of-the-declearn-api)
- - [Hands-on usage](#hands-on-usage)
- - [Overview of local differential-privacy capabilities](#local-differential-privacy)
- [Developers](#developers)
- [Copyright](#copyright)
--------------------
## Introduction
-`declearn` is a python package providing with a framework to perform federated
+[declearn](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/)
+is a python package providing with a framework to perform federated
learning, i.e. to train machine learning models by distributing computations
across a set of data owners that, consequently, only have to share aggregated
information (rather than individual data samples) with an orchestrating server
@@ -48,857 +45,40 @@ the use of a central agent.
## Setup
-### Requirements
+TL;DR:
+- Use `pip install declearn` to install the package's latest release from
+ [PyPI](https://pypi.org/project/declearn/).
+- Use `pip install declearn[all]` to install all extra dependencies, that
+ notably include network communication and machine learning frameworks.
+- You may be picky as to the extra dependencies you want to install: for
+ that, please have a look at the `[project.optional-dependencies]` section
+ of the [pyproject.toml](./pyproject.toml) file.
-- python >= 3.8
-- pip
-
-Third-party requirements are specified (and automatically installed) as part
-of the installation process, and may be consulted from the `pyproject.toml`
-file.
-
-### Optional requirements
-
-Some third-party requirements are optional, and may not be installed. These
-are also specified as part of the `pyproject.toml` file, and may be divided
-into two categories:<br/>
-(a) dependencies of optional, applied declearn components (such as the PyTorch
-and Tensorflow tensor libraries, or the gRPC and websockets network
-communication backends) that are not imported with declearn by default<br/>
-(b) dependencies for running tests on the package (mainly pytest and some of
-its plug-ins)
-
-The second category is more developer-oriented, while the first may or may not
-be relevant depending on the use case to which you wish to apply `declearn`.
-
-In the `pyproject.toml` file, the `[project.optional-dependencies]` tables
-`all` and `test` respectively list the first and (first + second) categories,
-while additional tables redundantly list dependencies unit by unit.
-
-### Using a virtual environment (optional)
-
-It is generally advised to use a virtual environment, to avoid any dependency
-conflict between declearn and packages you might use in separate projects. To
-do so, you may for example use python's built-in
-[venv](https://docs.python.org/3/library/venv.html), or the third-party tool
-[conda](https://docs.conda.io/en/latest/).
-
-Venv instructions (example):
-
-```bash
-python -m venv ~/.venvs/declearn
-source ~/.venvs/declearn/bin/activate
-```
-
-Conda instructions (example):
-
-```bash
-conda create -n declearn python=3.8 pip
-conda activate declearn
-```
-
-_Note: at the moment, conda installation is not recommended, because the
-package's installation is made slightly harder due to some dependencies being
-installable via conda while other are only available via pip/pypi, which caninstall
-lead to dependency-tracking trouble._
-
-### Installation
-
-#### Install from PyPI
-
-Stable releases of the package are uploaded to
-[PyPI](https://pypi.org/project/declearn/), enabling one to install with:
-
-```bash
-pip install declearn # optionally with version constraints and/or extras
-```
-
-#### Install from source
-
-Alternatively, to install from source, one may clone the git repository (or
-download the source code from a release) and run `pip install .` from its
-root folder.
-
-```bash
-git clone git@gitlab.inria.fr:magnet/declearn/declearn.git
-cd declearn
-pip install . # or pip install -e .
-```
-
-#### Install extra dependencies
-
-To also install optional requirements, add the name of the extras between
-brackets to the `pip install` command, _e.g._ running one of the following:
-
-```bash
-# Examples of cherry-picked installation instructions.
-pip install declearn[grpc] # install dependencies to use gRPC communications
-pip install declearn[torch] # install `declearn.model.torch` dependencies
-pip install declearn[tensorflow,torch] # install both tensorflow and torch
-
-# Instructions to install bundles of optional components.
-pip install declearn[all] # install all extra dependencies, save for testing
-pip install declearn[tests] # install all extra dependencies plus testing ones
-```
-
-#### Notes
-
-- If you are not using a virtual environment, select carefully the `pip` binary
- being called (e.g. use `python -m pip`), and/or add a `--user` flag to the
- pip command.
-- Developers may have better installing the package in editable mode, using
- `pip install -e .` from the repository's root folder.
-- If you are installing the package within a conda environment, it may be
- better to run `pip install --no-deps declearn` so as to only install the
- package, and then to manually install the dependencies listed in the
- `pyproject.toml` file, using `conda install` rather than `pip install`
- whenever it is possible.
+If you want to read more about how to setup for and finally install declearn,
+you may read the
+[installation guide](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/setup)
## Quickstart
-### Setting
-
-Here is a quickstart example on how to set up a federated learning process
-to learn a LASSO logistic regression model (using a scikit-learn backend)
-using pre-processed data, formatted as csv files with a "label" column,
-where each client has two files: one for training, the other for validation.
-
-Here, the code uses:
-
-- standard FedAvg strategy (SGD for local steps, averaging of updates weighted
- by clients' training dataset size, no modifications of server-side updates)
-- 10 rounds of training, with 5 local epochs performed at each round and
- 128-samples batch size
-- at least 1 and at most 3 clients, awaited for 180 seconds by the server
-- network communications using gRPC, on host "example.com" and port 8888
-
-Note that this example code may easily be adjusted to suit use cases, using
-other types of models, alternative federated learning algorithms and/or
-modifying the communication, training and validation hyper-parameters.
-Please refer to the [Hands-on usage](#hands-on-usage) section for a more
-detailed and general description of how to set up a federated learning
-task and process with declearn.
-
-### Server-side script
-
-```python
-import declearn
-
-model = declearn.model.sklearn.SklearnSGDModel.from_parameters(
- kind="classifier", loss="log_loss", penalty="l1"
-)
-netwk = declearn.communication.NetworkServerConfig(
- protocol="grpc", host="example.com", port=8888,
- certificate="path/to/certificate.pem",
- private_key="path/to/private_key.pem"
-)
-optim = declearn.main.config.FLOptimConfig.from_params(
- aggregator="averaging",
- client_opt=0.001,
-)
-server = declearn.main.FederatedServer(
- model, netwk, optim, checkpoint="outputs"
-)
-config = declearn.main.config.FLRunConfig.from_params(
- rounds=10,
- register={"min_clients": 1, "max_clients": 3, "timeout": 180},
- training={"n_epoch": 5, "batch_size": 128, "drop_remainder": False},
-)
-server.run(config)
-```
-
-### Client-side script
-
-```python
-import declearn
-
-netwk = declearn.communication.NetworkClientConfig(
- protocol="grpc",
- server_uri="example.com:8888",
- name="client_name",
- certificate="path/to/client_cert.pem"
-)
-train = declearn.dataset.InMemoryDataset(
- "path/to/train.csv", target="label",
- expose_classes=True # enable sharing of unique target values
-)
-valid = declearn.dataset.InMemoryDataset("path/to/valid.csv", target="label")
-client = declearn.main.FederatedClient(netwk, train, valid, checkpoint="outputs")
-client.run()
-```
-
-### Note on dependency sharing
-
-One important issue however that is not handled by declearn itself is that
-of ensuring that clients have loaded all dependencies that may be required
-to unpack the Model and Optimizer instances transmitted at initialization.
-At the moment, it is therefore left to users to agree on the dependencies
-that need to be imported as part of the client-side launching script.
-
-For example, if the trained model is an artificial neural network that uses
-PyTorch as implementation backend, clients will need to add the
-`import declearn.model.torch` statement in their code (and, obviously, to
-have `torch` installed). Similarly, if a custom declearn `OptiModule` was
-written to modify the way updates are computed locally by clients, it will
-need to be shared with clients - either as a package to be imported (like
-torch previously), or as a bit of source code to add on top of the script.
+Our [quickstart](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/quickstart)
+example is the right place to start with if you want too see in a glance
+what end-user declearn code looks like.
## Usage of the Python API
-### Overview of the Federated Learning process
-
-This overview describes the way the `declearn.main.FederatedServer`
-and `declearn.main.FederatedClient` pair of classes implement the
-federated learning process. It is however possible to subclass
-these and/or implement alternative orchestrating classes to define
-alternative overall algorithmic processes - notably by overriding
-or extending methods that define the sub-components of the process
-exposed here.
-
-#### Overall process orchestrated by the server
-
-- Initially:
- - have the clients connect and register for training
- - prepare model and optimizer objects on both sides
-- Iteratively:
- - perform a training round
- - perform an evaluation round
- - decide whether to continue, based on the number of
- rounds taken or on the evolution of the global loss
-- Finally:
- - restore the model weights that yielded the lowest global loss
- - notify clients that training is over, so they can disconnect
- and run their final routine (e.g. save the "best" model)
- - optionally checkpoint the "best" model
- - close the network server and end the process
-
-#### Detail of the process phases
-
-- **Registration process**:
- - Server:
- - open up registration (stop rejecting all received messages)
- - handle and respond to client-emitted registration requests
- - await criteria to have been met (exact or min/max number of clients
- registered, optionally under a given timeout delay)
- - close registration (reject future requests)
- - Client:
- - gather metadata about the local training dataset
- (_e.g._ dimensions and unique labels)
- - connect to the server and send a request to join training,
- including the former information
- - await the server's response (retry after a timeout if the request
- came in too soon, i.e. registration is not opened yet)
- - messaging : (JoinRequest <-> JoinReply)
-
-- **Post-registration initialization**
- - Server:
- - validate and aggregate clients-transmitted metadata
- - finalize the model's initialization using those metadata
- - send the model, local optimizer and evaluation metrics specs to clients
- - Client:
- - instantiate the model, optimizer and metrics based on server instructions
- - messaging: (InitRequest <-> GenericMessage)
-
-- **(Opt.) Local differential privacy initialization**
- - This step is optional; a flag in the InitRequest at the previous step
- indicates to clients that it is to happen, as a secondary substep.
- - Server:
- - send hyper-parameters to set up local differential privacy, including
- dp-specific hyper-parameters and information on the planned training
- - Client:
- - adjust the training process to use sample-wise gradient clipping and
- add gaussian noise to gradients, implementing the DP-SGD algorithm
- - set up a privacy accountant to monitor the use of the privacy budget
- - messaging: (PrivacyRequest <-> GenericMessage)
-
-- **Training round**:
- - Server:
- - select clients that are to participate
- - send data-batching and effort constraints parameters
- - send shared model weights and (opt. client-specific) auxiliary variables
- - Client:
- - update model weights and optimizer auxiliary variables
- - perform training steps based on effort constraints
- - step: compute gradients over a batch; compute updates; apply them
- - finally, send back local model weights and auxiliary variables
- - messaging: (TrainRequest <-> TrainReply)
- - Server:
- - unpack and aggregate clients' model weights into global updates
- - unpack and process clients' auxiliary variables
- - run global updates through the server's optimizer to modify and finally
- apply them
-
-- **Evaluation round**:
- - Server:
- - select clients that are to participate
- - send data-batching parameters and shared model weights
- - (_send effort constraints, unused for now_)
- - Client:
- - update model weights
- - perform evaluation steps based on effort constraints
- - step: update evaluation metrics, including the model's loss, over a batch
- - optionally checkpoint the model, local optimizer and evaluation metrics
- - send results to the server: optionally prevent sharing detailed metrics;
- always include the scalar validation loss value
- - messaging: (EvaluateRequest <-> EvaluateReply)
- - Server:
- - aggregate local loss values into a global loss metric
- - aggregate all other evaluation metrics and log their values
- - optionally checkpoint the model, optimizer, aggregated evaluation
- metrics and client-wise ones
-
-### Overview of the declearn API
-
-#### Package structure
-
-The package is organized into the following submodules:
-
-- `aggregator`:<br/>
-   Model updates aggregating API and implementations.
-- `communication`:<br/>
-   Client-Server network communications API and implementations.
-- `data_info`:<br/>
-   Tools to write and extend shareable metadata fields specifications.
-- `dataset`:<br/>
-   Data interfacing API and implementations.
-- `main`:<br/>
-   Main classes implementing a Federated Learning process.
-- `metrics`:<br/>
-   Iterative and federative evaluation metrics computation tools.
-- `model`:<br/>
-   Model interfacing API and implementations.
-- `optimizer`:<br/>
-   Framework-agnostic optimizer and algorithmic plug-ins API and tools.
-- `typing`:<br/>
-   Type hinting utils, defined and exposed for code readability purposes.
-- `utils`:<br/>
-   Shared utils used (extensively) across all of declearn.
-
-#### Main abstractions
-
-This section lists the main abstractions implemented as part of
-`declearn`, exposing their main object and usage, some examples
-of ready-to-use implementations that are part of `declearn`, as
-well as references on how to extend the support of `declearn`
-backend (notably, (de)serialization and configuration utils) to
-new custom concrete implementations inheriting the abstraction.
-
-- `declearn.model.api.Model`:
- - Object: Interface framework-specific machine learning models.
- - Usage: Compute gradients, apply updates, compute loss...
- - Examples:
- - `declearn.model.sklearn.SklearnSGDModel`
- - `declearn.model.tensorflow.TensorflowModel`
- - `declearn.model.torch.TorchModel`
- - Extend: use `declearn.utils.register_type(group="Model")`
-
-- `declearn.model.api.Vector`:
- - Object: Interface framework-specific data structures.
- - Usage: Wrap and operate on model weights, gradients, updates...
- - Examples:
- - `declearn.model.sklearn.NumpyVector`
- - `declearn.model.tensorflow.TensorflowVector`
- - `declearn.model.torch.TorchVector`
- - Extend: use `declearn.model.api.register_vector_type`
-
-- `declearn.optimizer.modules.OptiModule`:
- - Object: Define optimization algorithm bricks.
- - Usage: Plug into a `declearn.optimizer.Optimizer`.
- - Examples:
- - `declearn.optimizer.modules.AdagradModule`
- - `declearn.optimizer.modules.MomentumModule`
- - `declearn.optimizer.modules.ScaffoldClientModule`
- - `declearn.optimizer.modules.ScaffoldServerModule`
- - Extend:
- - Simply inherit from `OptiModule` (registration is automated).
- - To avoid it, use `class MyModule(OptiModule, register=False)`.
-
-- `declearn.optimizer.modules.Regularizer`:
- - Object: Define loss-regularization terms as gradients modifiers.
- - Usage: Plug into a `declearn.optimizer.Optimizer`.
- - Examples:
- - `declearn.optimizer.regularizer.FedProxRegularizer`
- - `declearn.optimizer.regularizer.LassoRegularizer`
- - `declearn.optimizer.regularizer.RidgeRegularizer`
- - Extend:
- - Simply inherit from `Regularizer` (registration is automated).
- - To avoid it, use `class MyRegularizer(Regularizer, register=False)`.
-
-- `declearn.metrics.Metric`:
- - Object: Define evaluation metrics to compute iteratively and federatively.
- - Usage: Compute local and federated metrics based on local data streams.
- - Examples:
- - `declearn.metric.BinaryRocAuc`
- - `declearn.metric.MeanSquaredError`
- - `declearn.metric.MuticlassAccuracyPrecisionRecall`
- - Extend:
- - Simply inherit from `Metric` (registration is automated).
- - To avoid it, use `class MyMetric(Metric, register=False)`
-
-- `declearn.communication.api.NetworkClient`:
- - Object: Instantiate a network communication client endpoint.
- - Usage: Register for training, send and receive messages.
- - Examples:
- - `declearn.communication.grpc.GrpcClient`
- - `declearn.communication.websockets.WebsocketsClient`
- - Extend:
- - Simply inherit from `NetworkClient` (registration is automated).
- - To avoid it, use `class MyClient(NetworkClient, register=False)`.
-
-- `declearn.communication.api.NetworkServer`:
- - Object: Instantiate a network communication server endpoint.
- - Usage: Receive clients' requests, send and receive messages.
- - Examples:
- - `declearn.communication.grpc.GrpcServer`
- - `declearn.communication.websockets.WebsocketsServer`
- - Extend:
- - Simply inherit from `NetworkServer` (registration is automated).
- - To avoid it, use `class MyServer(NetworkServer, register=False)`.
-
-- `declearn.dataset.Dataset`:
- - Object: Interface data sources agnostic to their format.
- - Usage: Yield (inputs, labels, weights) data batches, expose metadata.
- - Examples:
- - `declearn.dataset.InMemoryDataset`
- - Extend: use `declearn.utils.register_type(group="Dataset")`
-
-### Hands-on usage
-
-Here are details on how to set up server-side and client-side programs
-that will run together to perform a federated learning process. Generic
-remarks from the [Quickstart](#quickstart) section hold here as well, the
-former section being an overly simple exemplification of the present one.
-
-You can follow along on a concrete example that uses the UCI heart disease
-dataset, that is stored in the `examples/uci-heart` folder. You may refer
-to the `server.py` and `client.py` example scripts, that comprise comments
-indicating how the code relates to the steps described below. For further
-details on this example and on how to run it, please refer to its own
-`readme.md` file.
-
-#### Server setup instructions
-
-1. Define a Model:
-
- - Set up a machine learning model in a given framework
- (_e.g._ a `torch.nn.Module`).
- - Select the appropriate `declearn.model.api.Model` subclass to wrap it up.
- - Either instantiate the `Model` or provide a JSON-serialized configuration.
-
-2. Define a FLOptimConfig:
- - Select a `declearn.aggregator.Aggregator` (subclass) instance to define
- how clients' updates are to be aggregated into global-model updates on
- the server side.
- - Parameterize a `declearn.optimizer.Optimizer` (possibly using a selected
- pipeline of `declearn.optimizer.modules.OptiModule` plug-ins and/or a
- pipeline of `declearn.optimizer.regularizers.Regularizer` ones) to be
- used by clients to derive local step-wise updates from model gradients.
- - Similarly, parameterize an `Optimizer` to be used by the server to
- (optionally) refine the aggregated model updates before applying them.
- - Wrap these three objects into a `declearn.main.config.FLOptimConfig`,
- possibly using its `from_config` method to specify the former three
- components via configuration dicts rather than actual instances.
- - Alternatively, write up a TOML configuration file that specifies these
- components (note that 'aggregator' and 'server_opt' have default values
- and may therefore be left unspecified).
-
-3. Define a communication server endpoint:
-
- - Select a communication protocol (_e.g._ "grpc" or "websockets").
- - Select the host address and port to use.
- - Preferably provide paths to PEM files storing SSL-required information.
- - Wrap this into a config dict or use `declearn.communication.build_server`
- to instantiate a `declearn.communication.api.NetworkServer` to be used.
-
-4. Instantiate and run a FederatedServer:
-
- - Instantiate a `declearn.main.FederatedServer`:
- - Provide the Model, FLOptimConfig and Server objects or configurations.
- - Optionally provide a MetricSet object or its specs (i.e. a list of
- Metric instances, identifier names of (name, config) tuples), that
- defines metrics to be computed by clients on their validation data.
- - Optionally provide the path to a folder where to write output files
- (model checkpoints and global loss history).
- - Instantiate a `declearn.main.config.FLRunConfig` to specify the process:
- - Maximum number of training and evaluation rounds to run.
- - Registration parameters: exact or min/max number of clients to have
- and optional timeout delay spent waiting for said clients to join.
- - Training parameters: data-batching parameters and effort constraints
- (number of local epochs and/or steps to take, and optional timeout).
- - Evaluation parameters: data-batching parameters and effort constraints
- (optional maximum number of steps (<=1 epoch) and optional timeout).
- - Early-stopping parameters (optionally): patience, tolerance, etc. as
- to the global model loss's evolution throughout rounds.
- - Local Differential-Privacy parameters (optionally): (epsilon, delta)
- budget, type of accountant, clipping norm threshold, RNG parameters.
- - Alternatively, write up a TOML configuration file that specifies all of
- the former hyper-parameters.
- - Call the server's `run` method, passing it the former config object,
- the path to the TOML configuration file, or dictionaries of keyword
- arguments to be parsed into a `FLRunConfig` instance.
-
-
-#### Clients setup instructions
-
-1. Interface training data:
-
- - Select and parameterize a `declearn.dataset.Dataset` subclass that
- will interface the local training dataset.
- - Ensure its `get_data_specs` method exposes the metadata that is to
- be shared with the server (and nothing else, to prevent data leak).
-
-2. Interface validation data (optional):
+The [user guide](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/user-guide)
+is the natural entrypoint to learn more about declearn's take on Federated
+Learning, the current package capabilities, how to implement your own use
+case, and the structure and key points of the package's API.
- - Optionally set up a second Dataset interfacing a validation dataset,
- used in evaluation rounds. Otherwise, those rounds will be run using
- the training dataset - which can be slow and/or lead to overfitting.
-
-3. Define a communication client endpoint:
-
- - Select the communication protocol used (_e.g._ "grpc" or "websockets").
- - Provide the server URI to connect to.
- - Preferable provide the path to a PEM file storing SSL-required information
- (matching those used on the Server side).
- - Wrap this into a config dict or use `declearn.communication.build_client`
- to instantiate a `declearn.communication.api.NetworkClient` to be used.
-
-4. Run any necessary import statement:
-
- - If optional or third-party dependencies are known to be required, import
- them (_e.g._ `import declearn.model.torch`).
-
-5. Instantiate a FederatedClient and run it:
-
- - Instantiate a `declearn.main.FederatedClient`:
- - Provide the NetworkClient and Dataset objects or configurations.
- - Optionally specify `share_metrics=False` to prevent sharing evaluation
- metrics (apart from the aggregated loss) with the server out of privacy
- concerns.
- - Optionally provide the path to a folder where to write output files
- (model checkpoints and local loss history).
- - Call the client's `run` method and let the magic happen.
-
-#### Logging
-
-Note that this section and the quickstart example both left apart the option
-to configure logging associated with the federated client and server, and/or
-the network communication handlers they make use of. One may simply set up
-custom `logging.Logger` instances and pass them as arguments to the class
-constructors to replace the default, console-only, loggers.
-
-The `declearn.utils.get_logger` function may be used to facilitate the setup
-of such logger instances, defining their name, verbosity level, and whether
-messages should be logged to the console and/or to an output file.
-
-### Local Differential Privacy
-
-#### Basics
-
-`declearn` comes with the possibility to train models using local differential
-privacy, as described in the centralized case by Abadi et al, 2016,
-[Deep Learning with Differential Privacy](https://arxiv.org/abs/1607.00133).
-This means that training can provide per-client privacy guarantees with regard
-to the central server.
-
-In practice, this can be done by simply adding a privacy field to the config
-file, object or input dict to the `run` method of `FederatedServer`. Taking
-the Heart UCI example, one simply has one line to add to the server-side
-script (`examples/heart-uci/server.py`) in order to implement local DP,
-here using Renyi-DP with epsilon=5, delta=0.00001 and a sample-wise gradient
-clipping parameter that binds their euclidean norm below 3:
-
-```python
-# These are the last statements in the `run_server` function.
-run_cfg = FLRunConfig.from_params(
- # The following lines come from the base example:
- rounds=20,
- register={"min_clients": nb_clients},
- training={"batch_size": 30, "drop_remainder": False},
- evaluate={"batch_size": 50, "drop_remainder": False},
- early_stop={"tolerance": 0.0, "patience": 5, "relative": False},
- # DP-specific instructions (in their absence, do not use local DP):
- privacy={"accountant": "rdp", "budget": (5, 10e-5), "sclip_norm": 3},
-)
-server.run(run_cfg) # this is unaltered
-```
-
-This implementation can breach privacy garantees for some standard model
-architecture and training processes, see the _Warnings and limits_ section.
-
-#### More details on the backend
-
-Implementing local DP requires to change four key elements, which are
-automatically handled in declearn based on the provided privacy configuration:
-
-- **Add a privacy accountant**. We use the `Opacus` library, to set up a
-privacy accountant. The accountant is used in two key ways :
- - To calculate how much noise to add to the gradient at each trainig step
- to provide an $`(\epsilon-\delta)`$-DP guarantee over the total number of
- steps planned. This is where the heavily lifting is done, as estimating
- the tighest bounds on the privacy loss is a non-trivial problem. We default
- to the Renyi-DP accountant used in the original paper, but Opacus provides
- an evolving list of options, since this is an active area of research. For
- more details see the documentation of `declearn.main.utils.PrivacyConfig`.
- - To keep track of the privacy budget spent as training progresses, in
- particular in case of early stopping.
-
-- **Implement per-sample gradient clipping**. Clipping bounds the sensitivity
-of samples' contributions to model updates. It is performed using the
-`max_norm` parameter of `Model.compute_batch_gradients`.
-
-- **Implement noise-addition to applied gradients**. A gaussian noise with a
-tailored variance is drawn and added to the batch-averaged gradients based on
-which the local model is updated at each and every training step.
-
-- **Use Poisson sampling to draw batches**. This is done at the `Dataset` level
-using the `poisson` argument of `Dataset.generate_batches`.
- - As stated in the Opacus documentation, "Minibatches should be formed by
- uniform sampling, i.e. on each training step, each sample from the dataset
- is included with a certain probability p. Note that this is different from
- standard approach of dataset being shuffled and split into batches: each
- sample has a non-zero probability of appearing multiple times in a given
- epoch, or not appearing at all."
- - For more details, see Zhu and Wang, 2019,
- [Poisson Subsampled Renyi Differential Privacy](http://proceedings.mlr.press/v97/zhu19c/zhu19c.pdf)
-
-#### Warnings and limits
-
-Under certain model and training specifications, two silent breaches of formal
-privacy guarantees can occur. Some can be handled automatically if working
-with `torch`, but need to be manually checked for in other frameworks.
-
-- **Neural net layers that breach DP**. Standard architectures can lead
-to information leaking between batch samples. Know examples include batch
-normalization layers, LSTM, and multi-headed attention modules. In `torch`,
-checking a module for DP-compliance can be done using Opacus, by running:
-
- ```python
- #given an NN.module to be tested
- from opacus import PrivacyEngine
- dp_compatible_module = PrivacyEngine.get_compatible_module(module)
- ```
-
-- **Gradient accumulation**. This feature is not used in standard declearn
-models and training tools, but users that might try to write custom hacks
-to simulate large batches by setting a smaller batch size and executing the
-optimization step every N steps over the accumulated sum of output gradients
-should be aware that this is not compatible with Poisson sampling.
-
-Finally, note that at this stage the DP implementation in declearn is taken
-directly from the centralized training case, and as such does not account for
-nor make use of some specifities of the Federated Learning process, such as
-privacy amplification by iteration.
+To dive directly into the code's documentation, you may also jump to the
+[API reference](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/api).
## Developers
-### Contributions
-
-Contributions to `declearn` are welcome, whether to provide fixes, suggest
-new features (_e.g._ new subclasses of the core abstractions) or even push
-forward framework evolutions and API revisions.
-
-To contribute directly to the code (beyond posting issues on gitlab), please
-create a dedicated branch, and submit a **Merge Request** once you want your
-work reviewed and further processed to end up integrated into the main branch.
-
-The **git branching strategy** is the following:
-
-- The 'develop' branch is the main one and should receive all finalized changes
- to the source code. Release branches are then created and updated by cherry-
- picking from that branch. It therefore acts as a nightly stable version.
-- The 'rX.Y' branches are release branches for each and every X.Y versions.
- For past versions, these branches enable pushing patches towards a subminor
- version release (hence being version `X.Y.(Z+1)-dev`). For future versions,
- these branches enable cherry-picking commits from main to build up an alpha,
- beta, release-candidate and eventually stable `X.Y.0` version to release.
-- Feature branches should be created at will to develop features, enhancements,
- or even hotfixes that will later be merged into 'main' and eventually into
- one or multiple release branches.
-- It is legit to write up poc branches, as well as to split the development of
- a feature into multiple branches that will incrementally be merged into an
- intermediate feature branch that will eventually be merged into 'main'.
-
-The **coding rules** are fairly simple:
-
-- Abide by [PEP 8](https://peps.python.org/pep-0008/), in a way that is
- coherent with the practices already at work in declearn.
-- Abide by [PEP 257](https://peps.python.org/pep-0257/), _i.e._ write
- docstrings **everywhere** (unless inheriting from a method, the behaviour
- and signature of which are unmodified), again using formatting that is
- coherent with the declearn practices.
-- Type-hint the code, abiding by [PEP 484](https://peps.python.org/pep-0484/);
- note that the use of Any and of "type: ignore" comments is authorized, but
- should be remain sparse.
-- Lint your code with [mypy](http://mypy-lang.org/) (for static type checking)
- and [pylint](https://pylint.pycqa.org/en/latest/) (for more general linting);
- do use "type: ..." and "pylint: disable=..." comments where you think it
- relevant, preferably with some side explanations.
- (see dedicated sub-sections below: [pylint](#running-pylint-to-check-the-code)
- and [mypy](#running-mypy-to-type-check-the-code))
-- Reformat your code using [black](https://github.com/psf/black); do use
- (sparingly) "fmt: off/on" comments when you think it relevant
- (see dedicated sub-section [below](#running-black-to-format-the-code)).
-- Abide by [semver](https://semver.org/) when implementing new features or
- changing the existing APIs; try making changes non-breaking, document and
- warn about deprecations or behavior changes, or make a point for API-breaking
- changes, which we are happy to consider but might take time to be released.
-
-### Unit tests and code analysis
-
-Unit tests, as well as more-involved functional ones, are implemented under
-the `test/` folder of the present repository.
-They are implemented using the [PyTest](https://docs.pytest.org) framework,
-as well as some third-party plug-ins (refer to [Setup][#setup] for details).
-
-Additionally, code analysis tools are configured through the `pyproject.toml`
-file, and used to control code quality upon merging to the main branch. These
-tools are [black](https://github.com/psf/black) for code formatting,
-[pylint](https://pylint.pycqa.org/) for overall static code analysis and
-[mypy](https://mypy.readthedocs.io/) for static type-cheking.
-
-#### Running the test suite using tox
-
-The third-party [tox](https://tox.wiki/en/latest/) tools may be used to run
-the entire test suite within a dedicated virtual environment. Simply run `tox`
-from the commandline with the root repo folder as working directory. You may
-optionally specify the python version(s) with which you want to run tests.
-
-```bash
-tox # run with default python 3.8
-tox -e py310 # override to use python 3.10
-```
-
-Note that additional parameters for `pytest` may be passed as well, by adding
-`--` followed by any set of options you want at the end of the `tox` command.
-For example, to use the declearn-specific `--fulltest` option (see the section
-below), run:
-
-```bash
-tox [tox options] -- --fulltest
-```
-
-#### Running unit tests using pytest
-
-To run all the tests, simply use:
-
-```bash
-pytest test
-```
-
-To run the tests under a given module (here, "model"):
-
-```bash
-pytest test/model
-```
-
-To run the tests under a given file (here, "test_main.py"):
-
-```bash
-pytest test/test_main.py
-```
-
-Note that by default, some test scenarios that are considered somewhat
-superfluous~redundant will be skipped in order to save time. To avoid
-skipping these, and therefore run a more complete test suite, add the
-`--fulltest` option to pytest:
-
-```bash
-pytest --fulltest test # or any more-specific target you want
-```
-
-#### Running black to format the code
-
-The [black](https://github.com/psf/black) code formatter is used to enforce
-uniformity of the source code's formatting style. It is configured to have
-a maximum line length of 79 (as per [PEP 8](https://peps.python.org/pep-0008/))
-and ignore auto-generated protobuf files, but will otherwise modify files
-in-place when executing the following commands from the repository's root
-folder:
-
-```bash
-black declearn # reformat the package
-black test # reformat the tests
-```
-
-Note that it may also be called on individual files or folders.
-One may "blindly" run black, however it is actually advised to have a look
-at the reformatting operated, and act on any readability loss due to it. A
-couple of advice:
-
-1. Use `#fmt: off` / `#fmt: on` comments sparingly, but use them.
-<br/>It is totally okay to protect some (limited) code blocks from
-reformatting if you already spent some time and effort in achieving a
-readable code that black would disrupt. Please consider refactoring as
-an alternative (e.g. limiting the nest-depth of a statement).
-
-2. Pre-format functions and methods' signature to ensure style homogeneity.
-<br/>When a signature is short enough, black may attempt to flatten it as a
-one-liner, whereas the norm in declearn is to have one line per argument,
-all of which end with a trailing comma (for diff minimization purposes). It
-may sometimes be necessary to manually write the code in the latter style
-for black not to reformat it.
-
-Finally, note that the test suite run with tox comprises code-checking by
-black, and will fail if some code is deemed to require alteration by that
-tool. You may run this check manually:
-
-```bash
-black --check declearn # or any specific file or folder
-```
-
-#### Running pylint to check the code
-
-The [pylint](https://pylint.pycqa.org/) linter is expected to be used for
-static code analysis. As a consequence, `# pylint: disable=[some-warning]`
-comments can be found (and added) to the source code, preferably with some
-indication as to the rationale for silencing the warning (or error).
-
-A minimal amount of non-standard hyper-parameters are configured via the
-`pyproject.toml` file and will automatically be used by pylint when run
-from within the repository's folder.
-
-Most code editors enable integrating the linter to analyze the code as it is
-being edited. To lint the entire package (or some specific files or folders)
-one may simply run `pylint`:
-
-```bash
-pylint declearn # analyze the package
-pylint test # analyze the tests
-```
-
-Note that the test suite run with tox comprises the previous two commands,
-which both result in a score associated with the analyzed code. If the score
-does not equal 10/10, the test suite will fail - notably preventing acceptance
-of merge requests.
-
-#### Running mypy to type-check the code
-
-The [mypy](https://mypy.readthedocs.io/) linter is expected to be used for
-static type-checking code analysis. As a consequence, `# type: ignore` comments
-can be found (and added) to the source code, as sparingly as possible (mostly,
-to silence warnings about untyped third-party dependencies, false-positives,
-or locally on closure functions that are obvious enough to read from context).
-
-Code should be type-hinted as much and as precisely as possible - so that mypy
-actually provides help in identifying (potential) errors and mistakes, with
-code clarity as final purpose, rather than being another linter to silence off.
-
-A minimal amount of parameters are configured via the `pyproject.toml` file,
-and some of the strictest rules are disabled as per their default value (e.g.
-Any expressions are authorized - but should be used sparingly).
-
-Most code editors enable integrating the linter to analyze the code as it is
-being edited. To lint the entire package (or some specific files or folders)
-one may simply run `mypy`:
-
-```bash
-mypy declearn
-```
-
-Note that the test suite run with tox comprises the previous command. If mypy
-identifies errors, the test suite will fail - notably preventing acceptance
-of merge requests.
-
+Information for developers, such as how to contribute, coding rules, and how
+to run the tests, can be found in the
+[developer guide](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/devs-guide/).
## Copyright
diff --git a/declearn/__init__.py b/declearn/__init__.py
index e35504e03adf0b9a7edb07e034a50b2cc8e42a7a..52b6216b168251871dc53b1fe2aa7e9d4436b255 100644
--- a/declearn/__init__.py
+++ b/declearn/__init__.py
@@ -29,25 +29,26 @@ interfaces that cover some of the most popular frameworks, such as
Scikit-Learn, TensorFlow and PyTorch.
The package is organized into the following submodules:
-* aggregator:
+
+* [aggregator][declearn.aggregator]:
Model updates aggregating API and implementations.
-* communication:
+* [communication][declearn.communication]:
Client-Server network communications API and implementations.
-* data_info:
+* [data_info][declearn.data_info]:
Tools to write and extend shareable metadata fields specifications.
-* dataset:
+* [dataset][declearn.dataset]:
Data interfacing API and implementations.
-* main:
+* [main][declearn.main]:
Main classes implementing a Federated Learning process.
-* metrics:
+* [metrics][declearn.metrics]:
Iterative and federative evaluation metrics computation tools.
-* model:
+* [model][declearn.model]:
Model interfacing API and implementations.
-* optimizer:
+* [optimizer][declearn.optimizer]:
Framework-agnostic optimizer and algorithmic plug-ins API and tools.
-* typing:
+* [typing][declearn.typing]:
Type hinting utils, defined and exposed for code readability purposes.
-* utils:
+* [utils][declearn.utils]:
Shared utils used (extensively) across all of declearn.
"""
diff --git a/declearn/aggregator/__init__.py b/declearn/aggregator/__init__.py
index 2d5dbd9bec6e441664654539fb90eab67999133c..76c285654fb199870664407309608b7e8a3e9574 100644
--- a/declearn/aggregator/__init__.py
+++ b/declearn/aggregator/__init__.py
@@ -24,9 +24,13 @@ be used as "gradients" by the server's Optimizer to update the global
model.
This declearn submodule provides with:
-* Aggregator : abstract class defining an API for Vector aggregation
-* AveragingAggregator : average-based-aggregation Aggregator subclass
-* GradientMaskedAveraging : gradient Masked Averaging Aggregator subclass
+
+* [Aggregator][declearn.aggregator.Aggregator]:
+ abstract class defining an API for Vector aggregation
+* [AveragingAggregator][declearn.aggregator.AveragingAggregator]:
+ average-based-aggregation Aggregator subclass
+* [GradientMaskedAveraging][declearn.aggregator.GradientMaskedAveraging]:
+ gradient Masked Averaging Aggregator subclass
"""
from ._api import Aggregator
diff --git a/declearn/aggregator/_api.py b/declearn/aggregator/_api.py
index ee832d9348b7a20c04daaf4fe13d07c186e52e81..3711dda11e69d6d8771970ce1bc9f344cbb98cc4 100644
--- a/declearn/aggregator/_api.py
+++ b/declearn/aggregator/_api.py
@@ -50,19 +50,19 @@ class Aggregator(metaclass=ABCMeta):
The following attribute and method require to be overridden
by any non-abstract child class of `Aggregator`:
- name: str class attribute
+ - name: str class attribute
Name identifier of the class (should be unique across existing
Aggregator classes). Also used for automatic types-registration
of the class (see `Inheritance` section below).
- aggregate(updates: Dict[str, Vector], n_steps: Dict[str, int]) -> Vector:
+ - aggregate(updates: Dict[str, Vector], n_steps: Dict[str, int]) -> Vector:
Aggregate input vectors into a single one.
This is the main method for any `Aggregator`.
Overridable
-----------
- get_config() -> Dict[str, Any]:
+ - get_config() -> Dict[str, Any]:
Return a JSON-serializable configuration dict of an instance.
- from_config(Dict[str, Any]) -> Aggregator:
+ - from_config(Dict[str, Any]) -> Aggregator:
Classmethod to instantiate an Aggregator from a config dict.
Inheritance
@@ -75,6 +75,7 @@ class Aggregator(metaclass=ABCMeta):
"""
name: ClassVar[str] = NotImplemented
+ """Name identifier of the class, unique across Aggregator classes."""
def __init_subclass__(
cls,
diff --git a/declearn/communication/__init__.py b/declearn/communication/__init__.py
index e455f62ad468ef44ce46afe581f26727e7a16a32..819f28e1ac409a4bbed79d2e21ca0b05404dc163 100644
--- a/declearn/communication/__init__.py
+++ b/declearn/communication/__init__.py
@@ -22,27 +22,35 @@ endpoints for federated learning processes, as well as suitable messages to
be transmitted, and the available communication protocols.
This module contains the following core submodules:
-* api:
+
+* [api][declearn.communication.api]:
Base API to define client- and server-side communication endpoints.
-* messaging:
+* [messaging][declearn.communication.messaging]:
Message dataclasses defining information containers to be exchanged
between communication endpoints.
-It also exposes the following core utility functions:
-* build_client:
+It also exposes the following core utility functions and dataclasses:
+
+* [build_client][declearn.communication.build_client]:
Instantiate a NetworkClient, selecting its subclass based on protocol name.
-* build_server:
+* [build_server][declearn.communication.build_server]:
Instantiate a NetworkServer, selecting its subclass based on protocol name.
-* list_available_protocols:
+* [list_available_protocols][declearn.communication.list_available_protocols]:
List the protocol names for which both a NetworkClient and NetworkServer
classes are registered (hence available to `build_client`/`build_server`).
+* [NetworkClientConfig][declearn.communication.NetworkClientConfig]:
+ TOML-parsable dataclass for network clients' instantiation.
+* [NetworkServerConfig][declearn.communication.NetworkServerConfig]:
+ TOML-parsable dataclass for network servers' instantiation.
+
Finally, it defines the following protocol-specific submodules, provided
the associated third-party dependencies are available:
-* grpc:
+
+* [grpc][declearn.communication.grpc]:
gRPC-based network communication endpoints.
Requires the `grpcio` and `protobuf` third-party packages.
-* websockets:
+* [websockets][declearn.communication.websockets]:
WebSockets-based network communication endpoints.
Requires the `websockets` third-party package.
"""
diff --git a/declearn/communication/_build.py b/declearn/communication/_build.py
index 62cdf6279ffbb34d359d7f2d16cf77f61e78925b..1552eba942e197f45ce9be44314c6e7d45517528 100644
--- a/declearn/communication/_build.py
+++ b/declearn/communication/_build.py
@@ -86,6 +86,11 @@ def build_client(
**kwargs:
Any valid additional keyword parameter may be passed as well.
Refer to the target `NetworkClient` subclass for details.
+
+ Returns
+ -------
+ client: NetworkClient
+ NetworkClient communication endpoint instance.
"""
protocol = protocol.strip().lower()
try:
@@ -138,6 +143,11 @@ def build_server(
**kwargs:
Any valid additional keyword parameter may be passed as well.
Refer to the target `NetworkServer` subclass for details.
+
+ Returns
+ -------
+ server: NetworkServer
+ NetworkServer communication endpoint instance.
"""
# inherited signature; pylint: disable=too-many-arguments
protocol = protocol.strip().lower()
@@ -162,18 +172,44 @@ BuildServerConfig = dataclass_from_func(build_server)
class NetworkClientConfig(BuildClientConfig, TomlConfig): # type: ignore
- """TOML-parsable dataclass for network clients' instantiation."""
+ """TOML-parsable dataclass for network clients' instantiation.
+
+ This class enables parsing and bundling arguments to the
+ [build_client][declearn.communication.build_client] function
+ from a TOML config file (using the `from_toml` classmethod)
+ or from keyword arguments (using the `from_kwargs` one), and
+ calling that function using the `build_client` method.
+ """
def build_client(self) -> NetworkClient:
- """Build a NetworkClient from the wrapped parameters."""
+ """Build a NetworkClient from the wrapped parameters.
+
+ Returns
+ -------
+ client: NetworkClient
+ NetworkClient communication endpoint instance.
+ """
return self.call()
class NetworkServerConfig(BuildServerConfig, TomlConfig): # type: ignore
- """TOML-parsable dataclass for network servers' instantiation."""
+ """TOML-parsable dataclass for network servers' instantiation.
+
+ This class enables parsing and bundling arguments to the
+ [build_server][declearn.communication.build_server] function
+ from a TOML config file (using the `from_toml` classmethod)
+ or from keyword arguments (using the `from_kwargs` one), and
+ calling that function using the `build_server` method.
+ """
def build_server(self) -> NetworkServer:
- """Build a NetworkServer from the wrapped parameters."""
+ """Build a NetworkServer from the wrapped parameters.
+
+ Returns
+ -------
+ server: NetworkServer
+ NetworkServer communication endpoint instance.
+ """
return self.call()
diff --git a/declearn/communication/api/__init__.py b/declearn/communication/api/__init__.py
index ad974025505b0e0e341b35504a725171874a71c2..a4e52e1a235c9ce8aa7a2bbf0dbd01bdbc7d02b9 100644
--- a/declearn/communication/api/__init__.py
+++ b/declearn/communication/api/__init__.py
@@ -19,7 +19,12 @@
This module provides `NetworkClient` and `NetworkServer`, two abstract
base classes that are to be used as network communication endpoints for
-federated learning processes.
+federated learning processes:
+
+* [NetworkClient][declearn.communication.api.NetworkClient]:
+ Abstract class defining an API for client-side communication endpoints.
+* [NetworkServer][declearn.communication.api.NetworkServer]:
+ Abstract class defining an API for server-side communication endpoints.
"""
from ._client import NetworkClient
diff --git a/declearn/communication/api/_client.py b/declearn/communication/api/_client.py
index a3e01d15aad417a4e94e76fa1cca74d2d1aee973..9bc5f03cd13571a7d0e4648a6dbfb9ec46ab93b0 100644
--- a/declearn/communication/api/_client.py
+++ b/declearn/communication/api/_client.py
@@ -50,6 +50,7 @@ class NetworkClient(metaclass=ABCMeta):
`NetworkClient` object, its `start` method must first be awaited
and conversely, its `stop` method should be awaited to close the
connection:
+ ```
>>> client = ClientSubclass("example.domain.com:8765", "name", "cert_path")
>>> await client.start()
>>> try:
@@ -57,12 +58,15 @@ class NetworkClient(metaclass=ABCMeta):
>>> ...
>>> finally:
>>> await client.stop()
+ ```
An alternative syntax to achieve the former is using the client
object as an asynchronous context manager:
+ ```
>>> async with ClientSubclass(...) as client:
>>> client.register(data_info)
>>> ...
+ ```
Note that a declearn `NetworkServer` manages an allow-list of
clients, which is defined during a registration phase of limited
@@ -72,6 +76,7 @@ class NetworkClient(metaclass=ABCMeta):
"""
protocol: ClassVar[str] = NotImplemented
+ """Protocol name identifier, unique across NetworkClient classes."""
def __init_subclass__(
cls,
@@ -180,7 +185,7 @@ class NetworkClient(metaclass=ABCMeta):
Raises
-------
- TypeError:
+ TypeError
If the server does not return a JoinReply message.
"""
reply = await self._send_message(JoinRequest(self.name, data_info))
@@ -229,10 +234,10 @@ class NetworkClient(metaclass=ABCMeta):
Raises
------
- RuntimeError:
+ RuntimeError
If the server emits an Error message in response to the
message sent.
- TypeError:
+ TypeError
If the server returns a non-Empty message.
Note
@@ -254,12 +259,17 @@ class NetworkClient(metaclass=ABCMeta):
async def check_message(self, timeout: Optional[int] = None) -> Message:
"""Retrieve the next message sent by the server.
+ Parameters
+ ----------
+ timeout: int or None, default=None
+ Optional timeout delay, after which the server will send an
+ Error message with `messaging.flags.CHECK_MESSAGE_TIMEOUT`
+ if no other message awaits collection by this client.
+
Returns
-------
- action: str
- Instruction for the client.
- params: dict
- Associated parameters, as a JSON-serializable dict.
+ message: Message
+ Message received from the server.
Note
----
diff --git a/declearn/communication/api/_server.py b/declearn/communication/api/_server.py
index 9d73c472db43d9cc80f3a8313979c7e9d6a823c1..9c48695a7bbdf11cf6bb06f1e1d34766220fec32 100644
--- a/declearn/communication/api/_server.py
+++ b/declearn/communication/api/_server.py
@@ -47,6 +47,7 @@ class NetworkServer(metaclass=ABCMeta):
to connect to the server via a `NetworkServer` object, its `start`
method must first be awaited, and conversely, its `stop` method
should be awaited to close the connection:
+ ```
>>> server = ServerSubclass(
... "example.domain.com", 8765, "cert_path", "pkey_path"
... )
@@ -56,12 +57,15 @@ class NetworkServer(metaclass=ABCMeta):
>>> ...
>>> finally:
>>> await server.stop()
+ ```
An alternative syntax to achieve the former is using the server
object as an asynchronous context manager:
+ ```
>>> async with ServerSubclass(...) as server:
>>> data_info = server.wait_for_clients(...)
>>> ...
+ ```
Note that a `NetworkServer` manages an allow-list of clients,
which is defined based on `NetworkClient.register(...)`-emitted
@@ -70,6 +74,7 @@ class NetworkServer(metaclass=ABCMeta):
"""
protocol: ClassVar[str] = NotImplemented
+ """Protocol name identifier, unique across NetworkServer classes."""
def __init_subclass__(
cls,
@@ -210,7 +215,7 @@ class NetworkServer(metaclass=ABCMeta):
Raises
------
- RuntimeError:
+ RuntimeError
If the number of registered clients does not abide by the
provided boundaries at the end of the process.
@@ -250,7 +255,7 @@ class NetworkServer(metaclass=ABCMeta):
Raises
------
- asyncio.TimeoutError:
+ asyncio.TimeoutError
If `timeout` is set and is reached while the message is
yet to be collected by at least one of the clients.
"""
@@ -281,7 +286,7 @@ class NetworkServer(metaclass=ABCMeta):
Raises
------
- asyncio.TimeoutError:
+ asyncio.TimeoutError
If `timeout` is set and is reached while the message is
yet to be collected by at least one of the clients.
"""
@@ -318,7 +323,7 @@ class NetworkServer(metaclass=ABCMeta):
Raises
------
- asyncio.TimeoutError:
+ asyncio.TimeoutError
If `timeout` is set and is reached while the message is
yet to be collected by the client.
"""
@@ -346,7 +351,7 @@ class NetworkServer(metaclass=ABCMeta):
Raises
------
- asyncio.TimeoutError:
+ asyncio.TimeoutError
If any of the clients has failed to deliver a message
before `timeout` was reached.
diff --git a/declearn/communication/api/_service.py b/declearn/communication/api/_service.py
index d0449e6698e5aa0fb1aa3f1047f7456f7feedf53..83b387329e1febeba18d7897aca98a3816f86043 100644
--- a/declearn/communication/api/_service.py
+++ b/declearn/communication/api/_service.py
@@ -266,7 +266,7 @@ class MessagesHandler:
Raises
------
- asyncio.TimeoutError:
+ asyncio.TimeoutError
If `timeout` is set and is reached while the message is
yet to be collected by the client.
@@ -336,7 +336,7 @@ class MessagesHandler:
Raises
------
- asyncio.TimeoutError:
+ asyncio.TimeoutError
If `timeout` is set and is reached while no message has
been received from the client.
@@ -385,7 +385,7 @@ class MessagesHandler:
Raises
------
- RuntimeError:
+ RuntimeError
If the number of registered clients does not abide by the
provided boundaries at the end of the process.
diff --git a/declearn/communication/grpc/__init__.py b/declearn/communication/grpc/__init__.py
index 598822c0d5f3dce444bc9ebaaa83bd95f1dcf7a9..4e6390e0488a8d4e9132a9db9fb59b4593442432 100644
--- a/declearn/communication/grpc/__init__.py
+++ b/declearn/communication/grpc/__init__.py
@@ -15,7 +15,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""gRPC implementation of network communication endpoints."""
+"""gRPC implementation of network communication endpoints.
+
+The classes implemented here are:
+
+* [GrpcClient][declearn.communication.grpc.GrpcClient]:
+ Client-side network communication endpoint implementation using gRPC.
+* [GrpcServer][declearn.communication.grpc.GrpcServer]:
+ Server-side network communication endpoint implementation using gRPC.
+
+The [protobufs][declearn.communication.grpc.protobufs] submodule is also
+exposed, that provides with backend code auto-generated from a protobuf
+file, and is not considered part of the declearn stable API.
+"""
from . import protobufs
from ._client import GrpcClient
diff --git a/declearn/communication/grpc/protobufs/__init__.py b/declearn/communication/grpc/protobufs/__init__.py
index a4085a746901b0906882744023bb4efbded7655a..049e61ba5f6d8df0edd85007b2d96158745ca629 100644
--- a/declearn/communication/grpc/protobufs/__init__.py
+++ b/declearn/communication/grpc/protobufs/__init__.py
@@ -18,12 +18,14 @@
"""Load the gRPC backend code auto-generated from "message.proto".
Instructions to re-generate the code:
+
* From the commandline, with 'protobufs' as working directory, run:
- python -m grpc_tools.protoc -I . --python_out=. \
- --grpc_python_out=. ./message.proto
- sed -i -E 's/^import.*_pb2/from . \0/' ./*_pb2*.py
+ - `python -m grpc_tools.protoc -I . --python_out=. \
+ --grpc_python_out=. ./message.proto`
+ - `sed -i -E 's/^import.*_pb2/from . \0/' ./*_pb2*.py`
+
* On MAC OS, replace the second command with:
- sed -i '' -E 's/^(import.*_pb2)/from . \1/' ./*_pb2*.py
+ - `sed -i '' -E 's/^(import.*_pb2)/from . \1/' ./*_pb2*.py`
"""
try:
diff --git a/declearn/communication/messaging/__init__.py b/declearn/communication/messaging/__init__.py
index 464024fd6402be43853e98d3d1ce075455e0e87e..7fbc9572134a7125829f049f46bf4ebb2ab22579 100644
--- a/declearn/communication/messaging/__init__.py
+++ b/declearn/communication/messaging/__init__.py
@@ -15,7 +15,33 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Submodule defining messaging containers and flags used in declearn."""
+"""Submodule defining messaging containers and flags used in declearn.
+
+The submodule exposes the [Message][declearn.communication.messaging.Message]
+abstract base dataclass, and its children:
+
+* [CancelTraining][declearn.communication.messaging.CancelTraining]
+* [Empty][declearn.communication.messaging.Empty]
+* [Error][declearn.communication.messaging.Error]
+* [GenericMessage][declearn.communication.messaging.GenericMessage]
+* [GetMessageRequest][declearn.communication.messaging.GetMessageRequest]
+* [EvaluationReply][declearn.communication.messaging.EvaluationReply]
+* [EvaluationRequest][declearn.communication.messaging.EvaluationRequest]
+* [InitRequest][declearn.communication.messaging.InitRequest]
+* [JoinReply][declearn.communication.messaging.JoinReply]
+* [JoinRequest][declearn.communication.messaging.JoinRequest]
+* [PrivacyRequest][declearn.communication.messaging.PrivacyRequest]
+* [StopTraining][declearn.communication.messaging.StopTraining]
+* [TrainReply][declearn.communication.messaging.TrainReply]
+* [TrainRequest][declearn.communication.messaging.TrainRequest]
+
+It also exposes the [parse_message_from_string]\
+[declearn.communication.messaging.parse_message_from_string]
+function to recover the structures above from a dump string.
+
+Finally, it exposes a set of [flags][declearn.communication.messaging.flags],
+as constant strings that may be used conventionally as part of messages.
+"""
from . import flags
from ._messages import (
diff --git a/declearn/communication/messaging/flags.py b/declearn/communication/messaging/flags.py
index c3e0dc0e3048703657e1cb1ab5efa9fc09a6e26c..43fd6df91bb5ba2b1df7c5af8695fbe370adadd2 100644
--- a/declearn/communication/messaging/flags.py
+++ b/declearn/communication/messaging/flags.py
@@ -15,7 +15,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Communication flags used in declearn communication backends."""
+"""Communication flags used in declearn communication backends.
+
+This module exposes the following flags, which are all str constants:
+
+- REGISTRATION_UNSTARTED
+- REGISTRATION_OPEN
+- REGISTRATION_CLOSED
+- REGISTERED_WELCOME
+- REGISTERED_ALREADY
+- CHECK_MESSAGE_TIMEOUT
+- INVALID_MESSAGE
+- REJECT_UNREGISTERED
+"""
# Registration flags.
diff --git a/declearn/communication/websockets/__init__.py b/declearn/communication/websockets/__init__.py
index 858f7d6a2cbdd1ca8998deea428a96c0357a54c0..e29734fc198fabdd0f932f660fa72193a244ffb5 100644
--- a/declearn/communication/websockets/__init__.py
+++ b/declearn/communication/websockets/__init__.py
@@ -15,7 +15,15 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""WebSockets implementation of network communication endpoints."""
+"""WebSockets implementation of network communication endpoints.
+
+The classes implemented here are:
+
+* [WebsocketsClient][declearn.communication.websockets.WebsocketsClient]:
+ Client-side network communication endpoint implementation using WebSockets.
+* [WebsocketsServer][declearn.communication.websockets.WebsocketsServer]:
+ Server-side network communication endpoint implementation using WebSockets.
+"""
from ._client import WebsocketsClient
from ._server import WebsocketsServer
diff --git a/declearn/data_info/__init__.py b/declearn/data_info/__init__.py
index bc3d471d67a39f9ebd81aaf3c30b9f7ba74ce73a..fdc89dbf99e424bbc8dc50aa7a8dda4168f9199f 100644
--- a/declearn/data_info/__init__.py
+++ b/declearn/data_info/__init__.py
@@ -28,24 +28,29 @@ writing specifications for expected 'data_info' fields, and automating
their use to validate and combine individual 'data_info' dicts into an
aggregated one.
-DataInfoField API tools:
-* DataInfoField:
+DataInfoField API tools
+-----------------------
+
+* [DataInfoField][declearn.data_info.DataInfoField]:
Abstract class defining an API to write field-wise specifications.
-* register_data_info_field:
+* [register_data_info_field][declearn.data_info.register_data_info_field]:
Decorator for DataInfoField subclasses, enabling their effective use.
-* aggregate_data_info:
+* [aggregate_data_info][declearn.data_info.aggregate_data_info]:
Turn a list of individual 'data_info' dicts into an aggregated one.
-* get_data_info_fields_documentation:
+* [get_data_info_fields_documentation]\
+[declearn.data_info.get_data_info_fields_documentation]:
Gather documentation for all fields that have registered specs.
-Field specifications:
-* ClassesField:
+Field specifications
+--------------------
+
+* [ClassesField][declearn.data_info.ClassesField]:
Specification for the 'classes' field.
-* InputShapeField:
+* [InputShapeField][declearn.data_info.InputShapeField]:
Specification for the 'input_shape' field.
-* NbFeaturesField:
+* [NbFeaturesField][declearn.data_info.NbFeaturesField]:
Specification for the 'n_features' field.
-* NbSamplesField:
+* [NbSamplesField][declearn.data_info.NbSamplesField]:
Specification for the 'n_samples' field.
"""
diff --git a/declearn/data_info/_base.py b/declearn/data_info/_base.py
index d8cdb322177e7c940def738c899107619fd5c018..b45418fef65e9345b910fa0d96b670658ac891c8 100644
--- a/declearn/data_info/_base.py
+++ b/declearn/data_info/_base.py
@@ -163,16 +163,16 @@ def aggregate_data_info(
Raises
------
- KeyError:
+ KeyError
If a field in `required_fields` is missing from at least one
dict included in `clients_data_info`.
- ValueError:
+ ValueError
If any value of a shared field is invalid, or if values from
a field are incompatible for combination.
Warns
-----
- UserWarning:
+ UserWarning
If one of the return fields has no corresponding registered
`DataInfoField` specification class.
In that case, return the list of individual values in lieu
diff --git a/declearn/dataset/__init__.py b/declearn/dataset/__init__.py
index 62c7eb5f082a68e349f1b04520e801248333d57c..dc5759a4a714b836e3bf7348d27e48b79d98468d 100644
--- a/declearn/dataset/__init__.py
+++ b/declearn/dataset/__init__.py
@@ -22,12 +22,13 @@ data samples and key metadata while remaining agnostic of the way the data is
actually being loaded (from a source file, a database, another API...).
This declearn submodule provides with:
-* Dataset : abstract class defining an API to access training or testing data
-* InMemoryDataset : Dataset subclass serving numpy(-like) memory-loaded data
+
+* [Dataset][declearn.dataset.Dataset]:
+ abstract class defining an API to access training or testing data
+* [InMemoryDataset][declearn.dataset.InMemoryDataset]:
+ Dataset subclass serving numpy(-like) memory-loaded data
arrays
"""
from ._base import Dataset, DataSpecs, load_dataset_from_json
-
-# from ._sparse import sparse_from_file, sparse_to_file
from ._inmemory import InMemoryDataset
diff --git a/declearn/dataset/_inmemory.py b/declearn/dataset/_inmemory.py
index ddf88a3bfcfb8b1076e2e3154ee30969fd74060c..1d51f9af41e2adc2b706518ae4d1566feb901949 100644
--- a/declearn/dataset/_inmemory.py
+++ b/declearn/dataset/_inmemory.py
@@ -46,6 +46,7 @@ class InMemoryDataset(Dataset):
"""Dataset subclass serving numpy(-like) memory-loaded data arrays.
This subclass implements:
+
* yielding (X, [y], [w]) batches matching the scikit-learn API,
with data wrapped as numpy arrays, scipy sparse matrices,
or pandas dataframes (or series for y and w)
@@ -54,8 +55,7 @@ class InMemoryDataset(Dataset):
formats
Note: future code refactoring may divide these functionalities
- into two distinct base classes to articulate back into
- this one.
+ into two distinct base classes to articulate back into this one.
Attributes
----------
@@ -199,16 +199,17 @@ class InMemoryDataset(Dataset):
) -> DataArray:
"""Load a data array from a dump file.
- Supported file extensions are:
- .csv:
+ Supported file extensions
+ -------------------------
+ - `.csv`:
csv file, comma-delimited by default.
Any keyword arguments to `pandas.read_csv` may be passed.
- .npy:
+ - `.npy`:
Non-pickle numpy array dump, created with `numpy.save`.
- .sparse:
+ - `.sparse`:
Scipy sparse matrix dump, created with the custom
`declearn.data.sparse.sparse_to_file` function.
- .svmlight:
+ - `.svmlight`:
SVMlight sparse matrix and labels array dump.
Parse using `sklearn.load_svmlight_file`, and
return either features or labels based on the
@@ -232,10 +233,11 @@ class InMemoryDataset(Dataset):
Raises
------
- TypeError:
+ TypeError
If `path` is of unsupported extension.
- Any exception raised by data-loading functions may also be
- raised (e.g. if the file cannot be proprely parsed).
+
+ Any exception raised by data-loading functions may also be raised
+ (e.g. if the file cannot be proprely parsed).
"""
ext = os.path.splitext(path)[1]
if ext == ".csv":
@@ -256,13 +258,14 @@ class InMemoryDataset(Dataset):
) -> str:
"""Save a data array to a dump file.
- Supported types of data arrays are:
- pandas.DataFrame or pandas.Series:
- Dump to a comma-separated ".csv" file.
- numpy.ndarray:
- Dump to a non-pickle ".npy" file.
- scipy.sparse.spmatrix:
- Dump to a ".sparse" file, using a custom format
+ Supported types of data arrays
+ ------------------------------
+ - `pandas.DataFrame` or `pandas.Series`:
+ Dump to a comma-separated `.csv` file.
+ - `numpy.ndarray`:
+ Dump to a non-pickle `.npy` file.
+ - `scipy.sparse.spmatrix`:
+ Dump to a `.sparse` file, using a custom format
and `declearn.data.sparse.sparse_to_file`.
Parameters
@@ -284,7 +287,7 @@ class InMemoryDataset(Dataset):
Raises
------
- TypeError:
+ TypeError
If `array` is of unsupported type.
"""
# Select a file extension and set up the array-dumping function.
@@ -458,9 +461,11 @@ class InMemoryDataset(Dataset):
weights: data array or None
Optional sample weights; scikit-learn's `sample_weight`.
- Note: in this context, a 'data array' is either a numpy array,
- scipy sparse matrix, pandas dataframe or pandas series.
- Note: batched arrays are aligned along their first axis.
+ Notes
+ -----
+ - In this context, a 'data array' is either a numpy array,
+ scipy sparse matrix, pandas dataframe or pandas series.
+ - Batched arrays are aligned along their first axis.
"""
if poisson:
order = self._poisson_sampling(batch_size, drop_remainder)
@@ -582,7 +587,10 @@ class InMemoryDataset(Dataset):
Array containing a pre-computed samples' ordering.
Yield batches of samples drawn in that order from `data`.
- Yield slices of `data`, or None values if `data` is None.
+ Yields
+ ------
+ batch: optional data array
+ Slice of `data`, or None if `data` is None.
"""
if data is None:
yield from (None for _ in range(0, len(order), batch_size))
diff --git a/declearn/dataset/_sparse.py b/declearn/dataset/_sparse.py
index cbf8cbf4ab251fdaf2b90b898acb5679b5ee0e72..355537e83c3439affa46fc91e3cfe9d850aba7de 100644
--- a/declearn/dataset/_sparse.py
+++ b/declearn/dataset/_sparse.py
@@ -81,8 +81,9 @@ def sparse_to_file(
Raises
------
- TypeError if 'matrix' is of unsupported type, i.e. not
- a BSR, CSC, CSR, COO, DIA, DOK or LIL sparse matrix.
+ TypeError
+ If 'matrix' is of unsupported type, i.e. not a BSR,
+ CSC, CSR, COO, DIA, DOK or LIL sparse matrix.
Note: the format used is mostly similar to the SVMlight one
(see for example `sklearn.datasets.dump_svmlight_file`), but
@@ -130,10 +131,10 @@ def sparse_from_file(path: str) -> spmatrix:
Raises
------
- KeyError:
+ KeyError
If the file's header cannot be JSON-parsed or does not
conform to the expected standard.
- TypeError:
+ TypeError
If the documented sparse matrix type is not supported,
i.e. "bsr", "csv", "csc", "coo", "dia", "dok" or "lil".
diff --git a/declearn/main/__init__.py b/declearn/main/__init__.py
index 298900f961bd54418cdc5e081623da7f85771c26..49164f430f19ad0b92757fb122b139b7acd8d1dc 100644
--- a/declearn/main/__init__.py
+++ b/declearn/main/__init__.py
@@ -18,21 +18,25 @@
"""Main classes implementing a Federated Learning process.
This module mainly implements the following two classes:
-* FederatedClient: Client-side main Federated Learning orchestrating class
-* FederatedServer: Server-side main Federated Learning orchestrating class
+
+* [FederatedClient][declearn.main.FederatedClient]:
+ Client-side main Federated Learning orchestrating class
+* [FederatedServer][declearn.main.FederatedServer]:
+ Server-side main Federated Learning orchestrating class
This module also implements the following submodules, used by the former:
-* config:
+
+* [config][declearn.main.config]:
Server-side dataclasses that specify a FL process's parameter.
- The main class implemented here is `FLRunConfig`, that implements
- parameters' parsing from python objets or from a TOML config file.
-* privacy:
+ The main classes implemented here are `FLRunConfig` and `FLOptimConfig`,
+ that implement parameters' parsing from python objets or from TOML files.
+* [privacy][declearn.main.privacy]:
Differentially-Private training routine utils.
The main class implemented here is `DPTrainingManager` that implements
client-side DP-SGD training. This module is to be manually imported or
lazy-imported by `FederatedClient`, and may trigger warnings or errors
in the absence of the 'opacus' third-party dependency.
-* utils:
+* [utils][declearn.main.utils]:
Various utils to the FL process.
The main class of interest for end-users is `TrainingManager`, that
implements client-side training and evaluation routines, and may
diff --git a/declearn/main/_client.py b/declearn/main/_client.py
index cfd898fb784f9db7410fc3efd3ff126f2118843c..916e70b35f1b3b0674c2f887e11bde5e8ec7a67e 100644
--- a/declearn/main/_client.py
+++ b/declearn/main/_client.py
@@ -225,7 +225,7 @@ class FederatedClient:
Raises
------
- RuntimeError:
+ RuntimeError
If initialization failed, either because the message was not
received or was of incorrect type, or because instantiation
of the objects it specifies failed.
diff --git a/declearn/main/_server.py b/declearn/main/_server.py
index aa81783d011f42ae35ee57a331ce448d0810bf78..a33e1c80a465766e95f0ba4540ddc3f6cd9dde6a 100644
--- a/declearn/main/_server.py
+++ b/declearn/main/_server.py
@@ -231,7 +231,7 @@ class FederatedServer:
Raises
------
- RuntimeError:
+ RuntimeError
In case any of the clients returned an Error message rather
than an Empty ping-back message. Send CancelTraining to all
clients before raising.
@@ -281,7 +281,7 @@ class FederatedServer:
Raises
------
- AggregationError:
+ AggregationError
In case (some of) the clients' data info is invalid, or
incompatible. Send CancelTraining to all clients before
raising.
@@ -322,7 +322,7 @@ class FederatedServer:
Raises
------
- RuntimeError:
+ RuntimeError
If any client sent an incorrect message or reported
failure to conduct the evaluation step properly.
Send CancelTraining to all clients before raising.
diff --git a/declearn/main/config/__init__.py b/declearn/main/config/__init__.py
index 9114a6f627dfdc2c675467af5966f70bbe413fa0..c26bb7eef8287fac3cdfbad24678055809e9456c 100644
--- a/declearn/main/config/__init__.py
+++ b/declearn/main/config/__init__.py
@@ -23,22 +23,20 @@ are required to specify a Federated Learning process from the server's
side.
The main classes implemented here are:
-* FLRunConfig : federated learning orchestration hyper-parameters
-* FLOptimConfig : federated optimization strategy
-
-The following dataclasses are articulated by `FLRunConfig`:
-* EvaluateConfig : hyper-parameters for an evaluation round
-* RegisterConfig : hyper-parameters for clients registration
-* TrainingConfig : hyper-parameters for a training round
+* [FLRunConfig][declearn.main.config.FLRunConfig]:
+ Federated learning orchestration hyper-parameters.
+* [FLOptimConfig][declearn.main.config.FLOptimConfig]:
+ Federated optimization strategy.
-This submodule exposes dataclasses that group and document server-side
-hyper-parameters that specify a Federated Learning process, as well as
-a main class designed to act as a container and a parser for all these,
-that may be instantiated from python objects or from a TOML file.
+The following dataclasses are articulated by `FLRunConfig`:
-In other words, `FLRunConfig` in the key class implemented here, while
-the other exposed dataclasses are already articulated and used by it.
+* [EvaluateConfig][declearn.main.config.EvaluateConfig]:
+ Hyper-parameters for an evaluation round.
+* [RegisterConfig][declearn.main.config.RegisterConfig]:
+ Hyper-parameters for clients registration.
+* [TrainingConfig][declearn.main.config.TrainingConfig]:
+ Hyper-parameters for a training round.
"""
from ._dataclasses import (
diff --git a/declearn/main/config/_dataclasses.py b/declearn/main/config/_dataclasses.py
index cc3cd8b0f7cc455967fec5f0f31a16da878bc8c7..dc5174e9f3fb478242c4daf8d81a527f582158ed 100644
--- a/declearn/main/config/_dataclasses.py
+++ b/declearn/main/config/_dataclasses.py
@@ -150,11 +150,12 @@ class PrivacyConfig:
threshold, and RNG-related parameters for the noise-addition module.
Accountants supported by Opacus 1.2.0 include:
+
* rdp : Renyi-DP accountant, see [1]
* gdp : Gaussian-DP, see [2]
* prv : Privacy loss Random Variables privacy accountant, see [3]
- Note : for more details, refer to the Opacus source code and the
+ Note: for more details, refer to the Opacus source code and the
doctrings of each accountant. See
https://github.com/pytorch/opacus/tree/main/opacus/accountants
@@ -178,13 +179,16 @@ class PrivacyConfig:
References
----------
- [1] Abadi et al, 2016.
+ - [1]
+ Abadi et al, 2016.
Deep Learning with Differential Privacy.
https://arxiv.org/abs/1607.00133
- [2] Dong et al, 2019.
+ - [2]
+ Dong et al, 2019.
Gaussian Differential Privacy.
https://arxiv.org/abs/1905.02383
- [3] Gopi et al, 2021.
+ - [3]
+ Gopi et al, 2021.
Numerical Composition of Differential Privacy.
https://arxiv.org/abs/2106.02848
"""
diff --git a/declearn/main/config/_run_config.py b/declearn/main/config/_run_config.py
index 536157e3917bf336afe114a6be75274e11bd6506..ec68dbbd5353d59d13955547656e48bf3f9d42ac 100644
--- a/declearn/main/config/_run_config.py
+++ b/declearn/main/config/_run_config.py
@@ -56,39 +56,39 @@ class FLRunConfig(TomlConfig):
Fields
------
- rounds: int
+ - rounds: int
Maximum number of training and validation rounds to perform.
- register: RegisterConfig
+ - register: RegisterConfig
Parameters for clients' registration (min and/or max number
of clients to expect, optional max duration of the process).
- training: TrainingConfig
+ - training: TrainingConfig
Parameters for training rounds, including effort constraints
and data-batching instructions.
- evaluate: EvaluateConfig
+ - evaluate: EvaluateConfig
Parameters for validation rounds, similar to training ones.
- privacy: PrivacyConfig or None
+ - privacy: PrivacyConfig or None
Optional parameters to set up local differential privacy,
by having clients use the DP-SGD algorithm for training.
- early_stop: EarlyStopConfig or None
+ - early_stop: EarlyStopConfig or None
Optional parameters to set up an EarlyStopping criterion, to
be leveraged so as to interrupt the federated learning process
based on the tracking of a minimized quantity (e.g. model loss).
Instantiation classmethods
--------------------------
- from_toml:
+ - from_toml:
Instantiate by parsing a TOML configuration file.
- from_params:
+ - from_params:
Instantiate by parsing inputs dicts (or objects).
Notes
-----
- * `register` may be defined as a single integer (in `from_params` or in
+ - `register` may be defined as a single integer (in `from_params` or in
a TOML file), that will be used as the exact number of clients.
- * If `evaluate` is not provided to `from_params` or in the parsed TOML
+ - If `evaluate` is not provided to `from_params` or in the parsed TOML
file, default parameters will automatically be used and the training
batch size will be used for evaluation as well.
- * If `privacy` is provided and the 'poisson' parameter is unspecified
+ - If `privacy` is provided and the 'poisson' parameter is unspecified
for `training`, it will be set to True by default rather than False.
"""
@@ -108,6 +108,7 @@ class FLRunConfig(TomlConfig):
"""Field-specific parser to instantiate a RegisterConfig.
This method supports specifying `register`:
+
* as a single int, translated into {"min_clients": inputs}
* as None (or missing kwarg), using default RegisterConfig()
diff --git a/declearn/main/config/_strategy.py b/declearn/main/config/_strategy.py
index cfae32be56c782a1183f89ee243757593558664f..2fcd058e6eb8276ff16f26d786790ffc44a309de 100644
--- a/declearn/main/config/_strategy.py
+++ b/declearn/main/config/_strategy.py
@@ -49,43 +49,45 @@ class FLOptimConfig(TomlConfig):
Fields
------
- client_opt: Optimizer
+ - client_opt: Optimizer
Optimizer to be used by clients (that each hold a copy)
so as to conduct the step-wise local model updates.
- server_opt: Optimizer, default=Optimizer(lrate=1.0)
+ - server_opt: Optimizer, default=Optimizer(lrate=1.0)
Optimizer to be used by the server so as to conduct a
round-wise global model update based on the aggregated
client updates.
- aggregator: Aggregator, default=AverageAggregator()
+ - aggregator: Aggregator, default=AverageAggregator()
Client weights aggregator to be used by the server so as
to conduct the round-wise aggregation of client udpates.
Notes
-----
The `aggregator` field may be specified in a variety of ways:
+
- a single string may specify the registered name of the class
- constructor to use.
- In TOML, use `aggregator = "<name>"` outside of any section.
+ constructor to use.
+ In TOML, use `aggregator = "<name>"` outside of any section.
- a serialization dict, that specifies the registration `name`,
- and optionally a registration `group` and/or arguments to be
- passed to the class constructor.
- In TOML, use an `[aggregator]` section with a `name = "<name>"`
- field and any other fields you wish to pass. Kwargs may either
- be grouped into a dedicated `[aggregator.config]` sub-section
- or provided as fields of the main aggregator section.
+ and optionally a registration `group` and/or arguments to be
+ passed to the class constructor.
+ In TOML, use an `[aggregator]` section with a `name = "<name>"`
+ field and any other fields you wish to pass. Kwargs may either
+ be grouped into a dedicated `[aggregator.config]` sub-section
+ or provided as fields of the main aggregator section.
The `client_opt` and `server_opt` fields may be specified as:
+
- a single float, specifying the learning rate for vanilla SGD.
- In TOML, use `client_opt = 0.001` for `Optimizer(lrate=0.001)`.
+ In TOML, use `client_opt = 0.001` for `Optimizer(lrate=0.001)`.
- a dict of keyword arguments for `declearn.optimizer.Optimizer`.
- In TOML, use a `[client_opt]` section with fields specifying
- the input parameters you wish to pass to the constructor.
+ In TOML, use a `[client_opt]` section with fields specifying
+ the input parameters you wish to pass to the constructor.
Instantiation classmethods
--------------------------
- from_toml:
+ - from_toml:
Instantiate by parsing a TOML configuration file.
- from_params:
+ - from_params:
Instantiate by parsing inputs dicts (or objects).
"""
@@ -101,7 +103,7 @@ class FLOptimConfig(TomlConfig):
field: dataclasses.Field, # future: dataclasses.Field[Optimizer]
inputs: Union[float, Dict[str, Any], Optimizer],
) -> Optimizer:
- """ "Field-specific parser to instanciate the client-side Optimizer."""
+ """Field-specific parser to instantiate the client-side Optimizer."""
return cls._parse_optimizer(field, inputs)
@classmethod
@@ -110,7 +112,7 @@ class FLOptimConfig(TomlConfig):
field: dataclasses.Field, # future: dataclasses.Field[Optimizer]
inputs: Union[float, Dict[str, Any], Optimizer, None],
) -> Optimizer:
- """ "Field-specific parser to instanciate the server-side Optimizer."""
+ """Field-specific parser to instantiate the server-side Optimizer."""
return cls._parse_optimizer(field, inputs)
@classmethod
@@ -138,13 +140,14 @@ class FLOptimConfig(TomlConfig):
"""Field-specific parser to instantiate an Aggregator.
This method supports specifying `aggregator`:
- * as a str, used to retrieve a registered Aggregator class
- * as a dict, parsed a serialized Aggregator configuration:
+
+ - as a str, used to retrieve a registered Aggregator class
+ - as a dict, parsed a serialized Aggregator configuration:
- name: str used to retrieve a registered Aggregator class
- (opt.) group: str used to retrieve the registered class
- (opt.) config: dict specifying kwargs for the constructor
- any other field will be added to the `config` kwargs dict
- * as None (or missing kwarg), using default AverageAggregator()
+ - as None (or missing kwarg), using default AverageAggregator()
"""
# Case when using the default value: delegate to the default parser.
if inputs is None:
diff --git a/declearn/main/privacy/__init__.py b/declearn/main/privacy/__init__.py
index 99b325d1636c983e5885b9701dd293c6d040c3cb..aa256c7ffb56e80e0d915b068d2c3eed4a92d77c 100644
--- a/declearn/main/privacy/__init__.py
+++ b/declearn/main/privacy/__init__.py
@@ -15,6 +15,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Submodule implementing Differential-Privacy-oriented tools."""
+"""Submodule implementing Differential-Privacy-oriented tools.
+
+* [DPTrainingManager][declearn.main.privacy.DPTrainingManager]:
+ TrainingManager subclass implementing Differential Privacy mechanisms.
+"""
from ._dp_trainer import DPTrainingManager
diff --git a/declearn/main/privacy/_dp_trainer.py b/declearn/main/privacy/_dp_trainer.py
index 76b9f92ab8afd9d434d6a95f66adccd86dd078b4..fb79e7837b467cbedfa9d730da08829bc288f179 100644
--- a/declearn/main/privacy/_dp_trainer.py
+++ b/declearn/main/privacy/_dp_trainer.py
@@ -42,6 +42,7 @@ class DPTrainingManager(TrainingManager):
"""TrainingManager subclass adding Differential Privacy mechanisms.
This class extends the base TrainingManager class in three key ways:
+
* Perform per-sample gradients clipping (through the Model API),
parametrized by the added, optional `sclip_norm` attribute.
* Add noise to batch-averaged gradients at each step of training,
@@ -55,6 +56,12 @@ class DPTrainingManager(TrainingManager):
stochastic gradient descent algorithm (DP-SGD) [1] algorithm, in a
modular fashion that enables using any kind of optimizer plug-in
supported by its (non-DP) parent.
+
+ References
+ ----------
+ [1] Abadi et al, 2016.
+ Deep Learning with Differential Privacy.
+ https://arxiv.org/abs/1607.00133
"""
def __init__(
@@ -78,7 +85,14 @@ class DPTrainingManager(TrainingManager):
self,
message: messaging.PrivacyRequest,
) -> None:
- """Set up the use of DP-SGD based on a received PrivacyRequest."""
+ """Set up the use of DP-SGD based on a received PrivacyRequest.
+
+ Parameters
+ ----------
+ message: PrivacyRequest
+ PrivacyRequest message specifying the privacy budget, type
+ of accountant and expected use of the training data.
+ """
# REVISE: add support for fixed requested noise multiplier
# Compute the noise multiplier to use based on the budget
# and the planned training duration and parameters.
@@ -149,14 +163,30 @@ class DPTrainingManager(TrainingManager):
)
def get_noise_multiplier(self) -> Optional[float]:
- """Return the noise multiplier used for DP-SGD, if any."""
+ """Return the noise multiplier used for DP-SGD, if any.
+
+ Returns
+ -------
+ noise_multiplier: float or None
+ Standard deviation of the gaussian noise-addition module
+ placed at the start of the wrapped optimizer's pipeline,
+ if one is indeed present.
+ """
if self.optim.modules:
if isinstance(self.optim.modules[0], GaussianNoiseModule):
return self.optim.modules[0].std / (self.sclip_norm or 1.0)
return None
def get_privacy_spent(self) -> Tuple[float, float]:
- """Return the (epsilon, delta) privacy budget used."""
+ """Return the (epsilon, delta) privacy budget spent so far.
+
+ Returns
+ -------
+ epsilon: float
+ epsilon component of the privacy budget spent.
+ delta: float
+ delta component of the privacy budget spent.
+ """
if self.accountant is None:
raise RuntimeError("Cannot return spent privacy: DP is not used.")
delta = self._dp_budget[1]
diff --git a/declearn/main/utils/__init__.py b/declearn/main/utils/__init__.py
index 2fe4cb3c3fe1a5c88e4010987e0fd3ee03a619a7..5b903e43cd7b249c401b293a477e103ee86a9330 100644
--- a/declearn/main/utils/__init__.py
+++ b/declearn/main/utils/__init__.py
@@ -15,7 +15,48 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Utils for the main federated learning traning and evaluation processes."""
+"""Utils for the main federated learning traning and evaluation processes.
+
+TrainingManager
+---------------
+The main class implemented here is `TrainingManager`, that is used by clients
+and may also be used to perform centralized machine learning using declearn:
+
+* [TrainingManager][declearn.main.utils.TrainingManager]:
+
+End-user utils
+--------------
+Utils that may be composed into the main orchestration classes:
+
+* [Checkpointer][declearn.main.utils.Checkpointer]:
+ Model, optimizer, and metrics checkpointing class.
+* [EarlyStopping][declearn.main.utils.EarlyStopping]:
+ Class implementing a metric-based early-stopping decision rule.
+* [EarlyStopConfig][declearn.main.utils.EarlyStopConfig]:
+ Dataclass for EarlyStopping instantiation parameters.
+
+Backend: data-info aggregation
+------------------------------
+Backend utils to aggregate clients' dataset information:
+
+* [aggregate_clients_data_info]\
+[declearn.main.utils.aggregate_clients_data_info]:
+ Validate and aggregate clients' data-info dictionaries.
+* [AggregationError][declearn.main.utils.AggregationError]:
+ Custom exception that may be raised by `aggregate_clients_data_info`.
+
+Backend: effort constraints
+---------------------------
+Backend utils that are used to specify and articulate effort constraints
+for training and evaluation rounds:
+
+* [Constraint][declearn.main.utils.Constraint]:
+ Base class to implement effort constraints.
+* [ConstraintSet][declearn.main.utils.ConstraintSet]:
+ Utility class to wrap sets of Constraint instances.
+* [TimeoutConstraint][declearn.main.utils.TimeoutConstraint]:
+ Class implementing a simple time-based constraint.
+"""
from ._checkpoint import Checkpointer
from ._constraints import Constraint, ConstraintSet, TimeoutConstraint
diff --git a/declearn/main/utils/_checkpoint.py b/declearn/main/utils/_checkpoint.py
index 6d1a7a36c8190f1996ad50e016fd9b3a52dbea56..217fc1a48b013752ced71b9718205fd7de6d1ddc 100644
--- a/declearn/main/utils/_checkpoint.py
+++ b/declearn/main/utils/_checkpoint.py
@@ -108,8 +108,9 @@ class Checkpointer:
Raises
------
- TypeError:
+ TypeError
If `inputs` is of unproper type.
+
Other exceptions may be raised when calling this class's `__init__`.
"""
if isinstance(inputs, str):
diff --git a/declearn/main/utils/_data_info.py b/declearn/main/utils/_data_info.py
index b4061e86fc1fe3673ee5cd71d65978621d998218..b43184667da40c222a78815f0cd9985e2dac6ec1 100644
--- a/declearn/main/utils/_data_info.py
+++ b/declearn/main/utils/_data_info.py
@@ -87,7 +87,7 @@ def aggregate_clients_data_info(
Raises
------
- AggregationError:
+ AggregationError
In case any error is raised when calling `aggregate_data_info`
on the input arguments.
diff --git a/declearn/main/utils/_early_stop.py b/declearn/main/utils/_early_stop.py
index 86a516e5a20a0df8614607f4ab4758273c582735..92155e4832127725a7845f3b1b625d112ea814bd 100644
--- a/declearn/main/utils/_early_stop.py
+++ b/declearn/main/utils/_early_stop.py
@@ -113,3 +113,9 @@ class EarlyStopping:
EarlyStopConfig = dataclass_from_init(EarlyStopping, name="EarlyStopConfig")
+"""Auto-generated dataclass matching `EarlyStopping.__init__` signature.
+
+See [`declearn.main.utils.EarlyStopping.__init__`][] for details.
+You may also refer to the dynamically-generated docs, accessible using
+`help(EarlyStopConfig)` in a python terminal.
+""" # for docs rendering
diff --git a/declearn/main/utils/_training.py b/declearn/main/utils/_training.py
index 39ab2c7e81cab671e883209174458c8764f59e01..bc8f96429ca28a6a5e4ad4031340b7b738dd9d00 100644
--- a/declearn/main/utils/_training.py
+++ b/declearn/main/utils/_training.py
@@ -54,8 +54,8 @@ class TrainingManager:
) -> None:
"""Instantiate the client-side training and evaluation process.
- Arguments
- ---------
+ Parameters
+ ----------
model: Model
Model instance that needs training and/or evaluating.
optim: Optimizer
@@ -255,7 +255,7 @@ class TrainingManager:
Raises
------
- StopIteration:
+ StopIteration
If this step is being cancelled and the training routine
in the context of which it is being called should stop.
"""
diff --git a/declearn/metrics/__init__.py b/declearn/metrics/__init__.py
index d4e9cf5ddbf1c071795964bdc20bee1ff18197c3..2dcf7343e32c2b51a51b446e8349b5b29369cacb 100644
--- a/declearn/metrics/__init__.py
+++ b/declearn/metrics/__init__.py
@@ -21,35 +21,43 @@ This module provides with Metric, an abstract base class that defines
an API to iteratively and/or federatively compute evaluation metrics,
as well as a number of concrete standard machine learning metrics.
-Abstractions:
-* Metric:
+Abstractions
+------------
+* [Metric][declearn.metrics.Metric]:
Abstract base class defining an API for metrics' computation.
-* MeanMetric:
+* [MeanMetric][declearn.metrics.MeanMetric]:
Abstract class that defines a template for simple scores' averaging.
-Utils:
-* MetricSet:
+Utils
+-----
+* [MetricSet][declearn.metrics.MetricSet]:
Wrapper to bind together an ensemble of Metric instances.
* MetricInputType:
Type alias for valid inputs to specify a metric for `MetricSet`.
Equivalent to `Union[Metric, str, Tuple[str, Dict[str, Any]]]`.
-Classification metrics:
-* BinaryAccuracyPrecisionRecall
+Classification metrics
+----------------------
+* [BinaryAccuracyPrecisionRecall]\
+[declearn.metrics.BinaryAccuracyPrecisionRecall]:
Accuracy, precision, recall and confusion matrix for binary classif.
Identifier name: "binary-classif".
-* MulticlassAccuracyPrecisionRecall
+* [MulticlassAccuracyPrecisionRecall]\
+[declearn.metrics.MulticlassAccuracyPrecisionRecall]:
Accuracy, precision, recall and confusion matrix for multiclass classif.
Identifier name: "multi-classif".
-* BinaryRocAuc:
+* [BinaryRocAUC][declearn.metrics.BinaryRocAUC]:
Receiver Operator Curve and its Area Under the Curve for binary classif.
- Identified name: "binary-roc"
+ Identifier name: "binary-roc".
-Regression metrics:
-* MeanAbsoluteError:
+Regression metrics
+------------------
+* [MeanAbsoluteError][declearn.metrics.MeanAbsoluteError]:
Mean absolute error, averaged across all samples (and channels).
-* MeanSquaredError:
+ Identifier name: "mae".
+* [MeanSquaredError][declearn.metrics.MeanSquaredError]:
Mean squared error, averaged across all samples (and channels).
+ Identifier name: "mse".
"""
from ._api import Metric
diff --git a/declearn/metrics/_api.py b/declearn/metrics/_api.py
index d19cc2f210c89f9ca7d8d7322b441445e297c768..ab87f2299deeb6062e7be9cda69bd13e19a3ed3b 100644
--- a/declearn/metrics/_api.py
+++ b/declearn/metrics/_api.py
@@ -44,13 +44,18 @@ class Metric(metaclass=ABCMeta):
results through iterative update steps that may additionally
be run in a federative way.
+ Usage
+ -----
Single-party usage:
+ ```
>>> metric = MetricSubclass()
>>> metric.update(y_true, y_pred) # take one update state
>>> metric.get_result() # after one or multiple updates
>>> metric.reset() # reset before a next evaluation round
+ ```
Multiple-parties usage:
+ ```
>>> # Instantiate 2+ metrics and run local update steps.
>>> metric_0 = MetricSubclass()
>>> metric_1 = MetricSubclass()
@@ -61,23 +66,24 @@ class Metric(metaclass=ABCMeta):
>>> metric_1.agg_states(states_0) # metrics_1 is updated
>>> # Compute results that aggregate info from both clients.
>>> metric_1.get_result()
+ ```
Abstract
--------
To define a concrete Metric, one must subclass it and define:
- name: str class attribute
+ - name: str class attribute
Name identifier of the class (should be unique across existing
Metric classes). Also used for automatic types-registration of
the class (see `Inheritance` section below).
- _build_states() -> dict[str, (float | np.ndarray)]:
+ - _build_states() -> dict[str, (float | np.ndarray)]:
Build and return an ensemble of state variables.
This method is called to initialize the `_states` attribute,
that should be used and updated by other abstract methods.
- update(y_true: np.ndarray, y_pred: np.ndarray, s_wght: (np.ndarray|None)):
+ - update(y_true: np.ndarray, y_pred: np.ndarray, s_wght: np.ndarray|None):
Update the metric's internal state based on a data batch.
This method should update `self._states` in-place.
- get_result() -> dict[str, (float | np.ndarray)]:
+ - get_result() -> dict[str, (float | np.ndarray)]:
Compute the metric(s), based on the current state variables.
This method should make use of `self._states` and prevent
side effects on its contents.
@@ -87,7 +93,7 @@ class Metric(metaclass=ABCMeta):
Some methods may be overridden based on the concrete Metric's needs.
The most imporant one is the states-aggregation method:
- agg_states(states: dict[str, (float | np.ndarray)]:
+ - agg_states(states: dict[str, (float | np.ndarray)]:
Aggregate provided state variables into self ones.
By default, it expects input and internal states to have
similar specifications, and aggregates them by summation,
@@ -96,18 +102,18 @@ class Metric(metaclass=ABCMeta):
A pair of methods may be extended to cover non-`self._states`-contained
variables:
- reset():
+ - reset():
Reset the metric to its initial state.
- get_states() -> dict[str, (float | np.ndarray)]:
+ - get_states() -> dict[str, (float | np.ndarray)]:
Return a copy of the current state variables.
Finally, depending on the hyper-parameters defined by the subclass's
`__init__`, one should adjust JSON-configuration-interfacing methods:
- get_config() -> dict[str, any]:
+ - get_config() -> dict[str, any]:
Return a JSON-serializable configuration dict for this Metric.
- from_config(config: dict[str, any]) -> Self:
+ - from_config(config: dict[str, any]) -> Self:
Instantiate a Metric from its configuration dict.
Inheritance
@@ -121,6 +127,7 @@ class Metric(metaclass=ABCMeta):
"""
name: ClassVar[str] = NotImplemented
+ """Name identifier of the class, unique across Metric classes."""
def __init__(
self,
@@ -239,11 +246,11 @@ class Metric(metaclass=ABCMeta):
Raises
------
- KeyError:
+ KeyError
If any state variable is missing from `states`.
- TypeError:
+ TypeError
If any state variable is of unproper type.
- ValueError:
+ ValueError
If any array state variable is of unproper shape.
"""
final = {} # type: Dict[str, Union[float, np.ndarray]]
@@ -304,7 +311,7 @@ class Metric(metaclass=ABCMeta):
Raises
------
- KeyError:
+ KeyError
If the provided `name` fails to be mapped to a registered
Metric subclass.
"""
diff --git a/declearn/metrics/_classif.py b/declearn/metrics/_classif.py
index 30a0f3a348709a16d3018b252c2a7c056ebe225f..35c30f3412d9b711d0e83e7cd41835f36b60bb58 100644
--- a/declearn/metrics/_classif.py
+++ b/declearn/metrics/_classif.py
@@ -40,6 +40,7 @@ class BinaryAccuracyPrecisionRecall(Metric):
time, from which basic evaluation metrics may be derived.
Computed metrics are the following:
+
* accuracy: float
Accuracy of the classifier, i.e. P(pred==true).
Formula: (TP + TN) / (TP + TN + FP + FN)
@@ -132,6 +133,7 @@ class MulticlassAccuracyPrecisionRecall(Metric):
one-hot encodings) may be passed as predictions.
Computed metrics are the following:
+
* accuracy: float
Overall accuracy of the classifier, i.e. P(pred==true).
* precision: 1-d numpy.ndarray
diff --git a/declearn/metrics/_mean.py b/declearn/metrics/_mean.py
index 7ac765a12c2309f18dfce7dcc9c1142d9310a619..5b505346de8c4d4c5ae69db4257308e571f0a87d 100644
--- a/declearn/metrics/_mean.py
+++ b/declearn/metrics/_mean.py
@@ -42,11 +42,11 @@ class MeanMetric(Metric, register=False, metaclass=ABCMeta):
--------
To implement such an actual Metric, inherit `MeanMetric` and define:
- name: str class attribute:
+ - name: str class attribute:
Name identifier of the Metric (should be unique across existing
Metric classes). Used for automated type-registration and name-
based retrieval. Also used to label output results.
- metric_func(y_true: np.ndarray, y_pred: np.ndarray) -> np.ndarray:
+ - metric_func(y_true: np.ndarray, y_pred: np.ndarray) -> np.ndarray:
Method that computes a score from the predictions and labels
associated with a given batch, that is to be aggregated into
an average metric across all input batches.
@@ -119,7 +119,8 @@ class MeanAbsoluteError(MeanMetric):
with multiple channels, the sum of absolute channel-wise errors
is computed for each sample, and averaged across samples.
- Computed metrics is the following:
+ Computed metric is the following:
+
* mae: float
Mean absolute error, averaged across samples (possibly
summed over channels for (>=2)-dimensional inputs).
@@ -147,7 +148,8 @@ class MeanSquaredError(MeanMetric):
with multiple channels, the sum of squared channel-wise errors
is computed for each sample, and averaged across samples.
- Computed metrics is the following:
+ Computed metric is the following:
+
* mse: float
Mean squared error, averaged across samples (possibly
summed over channels for (>=2)-dimensional inputs).
diff --git a/declearn/metrics/_roc_auc.py b/declearn/metrics/_roc_auc.py
index 4ed37a853cc57f5e12b99ed0ab8b5785d6712c4b..a8326ca2e3a6ef50aea168b6deb4f42999a6298b 100644
--- a/declearn/metrics/_roc_auc.py
+++ b/declearn/metrics/_roc_auc.py
@@ -40,6 +40,7 @@ class BinaryRocAUC(Metric):
AUC metrics are eventually derived.
Computed metrics are the following:
+
* fpr: 1-d numpy.ndarray
True-positive rate values for a variety of thresholds.
Formula: TP / (TP + FN), i.e. P(pred=1|true=1)
@@ -83,14 +84,14 @@ class BinaryRocAUC(Metric):
Notes
-----
- Using the default `bound=None` enables the thresholds at which
- the ROC curve points are compute to vary dynamically based on
- inputs, but also based on input states to the `agg_states`
- method, that may come from a metric with different parameters.
- Setting up explicit boundaries prevents thresholds from being
- adjusted at update time, and a ValueError will be raise by the
- `agg_states` method if inputs are adjusted to a distinct set
- of thresholds.
+ - Using the default `bound=None` enables the thresholds at which
+ the ROC curve points are compute to vary dynamically based on
+ inputs, but also based on input states to the `agg_states`
+ method, that may come from a metric with different parameters.
+ - Setting up explicit boundaries prevents thresholds from being
+ adjusted at update time, and a ValueError will be raise by the
+ `agg_states` method if inputs are adjusted to a distinct set
+ of thresholds.
"""
self.scale = scale
self.label = label
diff --git a/declearn/metrics/_wrapper.py b/declearn/metrics/_wrapper.py
index 7bee089d0e1b35841cf9bf4eb80eec3960409df5..990bfabc5ae79c0952ce962774a7ba0fa5a70fa3 100644
--- a/declearn/metrics/_wrapper.py
+++ b/declearn/metrics/_wrapper.py
@@ -64,11 +64,11 @@ class MetricSet:
Raises
------
- TypeError:
+ TypeError
If one of the input `metrics` elements is of improper type.
- KeyError:
+ KeyError
If a metric name identifier fails to be mapped to a Metric class.
- RuntimeError:
+ RuntimeError
If multiple metrics are of the same final type.
"""
# REVISE: store metrics into a Dict and adjust labels when needed
@@ -117,8 +117,9 @@ class MetricSet:
Raises
------
- TypeError:
+ TypeError
If `metrics` is of unproper type.
+
Other exceptions may be raised when calling this class's `__init__`.
"""
if metrics is None:
@@ -210,11 +211,11 @@ class MetricSet:
Raises
------
- KeyError:
+ KeyError
If any state variable is missing from `states`.
- TypeError:
+ TypeError
If any state variable is of unproper type.
- ValueError:
+ ValueError
If any array state variable is of unproper shape.
"""
for metric in self.metrics:
diff --git a/declearn/model/__init__.py b/declearn/model/__init__.py
index 08d9cdb897d87c7572a1eb21499697ff53d4b0d1..d24c21e6a870c20daaf42427eb8ec738a98a2e1e 100644
--- a/declearn/model/__init__.py
+++ b/declearn/model/__init__.py
@@ -18,9 +18,43 @@
"""Model interfacing submodule, defining an API an derived applications.
This declearn submodule provides with:
-* Model and Vector abstractions, used as an API to design FL algorithms
-* Submodules implementing interfaces to curretnly supported frameworks
-and models.
+
+- Model and Vector abstractions, used as an API to design FL algorithms.
+- Submodules implementing interfaces to various frameworks and models.
+
+Default Submodules
+------------------
+The automatically-imported submodules implemented here are:
+
+* [api][declearn.model.api]:
+ Model and Vector abstractions' defining module.
+ - [Model][declearn.model.api.Model]:
+ abstract API to interface framework-specific models.
+ - [Vector][declearn.model.api.Vector]:
+ abstract API for data tensors containers.
+* [sklearn][declearn.model.sklearn]:
+ Scikit-Learn based or oriented tools
+ - [NumpyVector][declearn.model.sklearn.NumpyVector]
+ Vector for numpy array data structures.
+ - [SklearnSGDModel][declearn.model.sklearn.SklearnSGDModel]
+ Model for scikit-learn's SGDClassifier and SGDRegressor.
+
+Optional Submodules
+-------------------
+The optional-dependency-based submodules that may be manually imported are:
+
+* [tensorflow][declearn.model.tensorflow]:
+ TensorFlow-interfacing tools
+ - [TensorflowModel][declearn.model.tensorflow.TensorflowModel]:
+ Model to wrap any tensorflow-keras Layer model.
+ - [TensorflowVector][declearn.model.tensorflow.TensorflowVector]:
+ Vector for tensorflow Tensor and IndexedSlices.
+* [torch][declearn.model.torch]:
+ PyTorch-interfacing tools
+ - [TorchModel][declearn.model.torch.TorchModel]:
+ Model to wrap any torch Module model.
+ - [TorchVector][declearn.model.torch.TorchVector]:
+ Vector for torch Tensor objects.
"""
from . import api
diff --git a/declearn/model/api/__init__.py b/declearn/model/api/__init__.py
index ab147758c5187ff8c62a507d3c63824424045646..888b4c4e70cd3bbd13b7385bd6f123b1befd7c32 100644
--- a/declearn/model/api/__init__.py
+++ b/declearn/model/api/__init__.py
@@ -15,13 +15,17 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Model Vector abstractions submodule."""
+"""Model and Vector abstractions submodule.
+
+This submodules exports the building blocks of the Model and Vector APIs:
+
+* [Model][declearn.model.api.Model]:
+ Abstract class defining an API to interface a ML model.
+* [Vector][declearn.model.api.Vector]:
+ Abstract class defining an API to manipulate (sets of) data arrays.
+* [register_vector_type][declearn.model.api.register_vector_type]:
+ Decorate a Vector subclass to make it buildable with `Vector.build`.
+"""
from ._vector import Vector, register_vector_type
from ._model import Model
-
-__all__ = [
- "Model",
- "Vector",
- "register_vector_type",
-]
diff --git a/declearn/model/api/_model.py b/declearn/model/api/_model.py
index 6bd5fa081494f3862affba1fdbf95b2919228ebd..06cf0ce56728d37ff1fc9ece573447dff687d8ed 100644
--- a/declearn/model/api/_model.py
+++ b/declearn/model/api/_model.py
@@ -59,8 +59,8 @@ class Model(metaclass=ABCMeta):
) -> Set[str]:
"""List of 'data_info' fields required to initialize this model.
- Note: these fields should match a registered specification
- (see `declearn.data_info` submodule)
+ Note: These fields should match a registered specification
+ (see the [`declearn.data_info`][] submodule).
"""
@abstractmethod
@@ -78,7 +78,7 @@ class Model(metaclass=ABCMeta):
Raises
------
- KeyError:
+ KeyError
If some fields in `required_data_info` are missing.
Notes
diff --git a/declearn/model/api/_vector.py b/declearn/model/api/_vector.py
index c55986773c830b5d0ddaecdee596879ee1efdfb5..6434760c1b608670fc2b6ae570b476bb79f9d764 100644
--- a/declearn/model/api/_vector.py
+++ b/declearn/model/api/_vector.py
@@ -51,6 +51,7 @@ class Vector(metaclass=ABCMeta):
Use `vector.coefs` to access the stored coefficients.
Any concrete Vector subclass should:
+
- add type checks to `__init__` to control wrapped coefficients' type
- opt. override `_op_...` properties to define compatible operators
- implement the abstract operators (`sign`, `maximum`, `minimum`...)
@@ -89,11 +90,13 @@ class Vector(metaclass=ABCMeta):
def compatible_vector_types(self) -> Set[Type["Vector"]]:
"""Compatible Vector types, that may be combined into this.
- Note that VectorTypeA may be compatible with VectorTypeB
- while the opposite is False. It means that, for example,
- (VectorTypeB + VectorTypeA) -> VectorTypeB
- while
- (VectorTypeA + VectorTypeB) -> TypeError
+ If VectorTypeA is listed as compatible with VectorTypeB,
+ then `(VectorTypeB + VectorTypeA) -> VectorTypeB` (both
+ for addition and any basic operator).
+
+ However, compatibility may not go both ways, so that for
+ the same types, `(VectorTypeA + VectorTypeB)` will raise
+ a TypeError.
This is for example the case is VectorTypeB stores numpy
arrays while VectorTypeA stores tensorflow tensors since
@@ -268,7 +271,8 @@ class Vector(metaclass=ABCMeta):
Function to be applied to each and every coefficient (data
array) wrapped by this Vector, that must return a similar
array (same type, shape and dtype).
- *args and **kwargs to `func` may also be passed.
+
+ Any `*args` and `**kwargs` to `func` may also be passed.
Returns
-------
@@ -429,9 +433,10 @@ def register_vector_type(
*types: Type[Any],
name: Optional[str] = None,
) -> Callable[[Type[Vector]], Type[Vector]]:
- """Decorate a Vector subclass to make it buildable with `Vector(...)`.
+ """Decorate a Vector subclass to make it buildable with `Vector.build`.
Decorating a Vector subclass with this has three effects:
+
* Add the class to registered type (in the "Vector" group).
See `declearn.utils.register_type` for details.
* Make instances of that class JSON-serializable, embarking
diff --git a/declearn/model/sklearn/__init__.py b/declearn/model/sklearn/__init__.py
index 15702ec3696479d790741d856fdc19d743748f7b..776b82ffc8696a9540bcf4b95c65f8ec7929ba64 100644
--- a/declearn/model/sklearn/__init__.py
+++ b/declearn/model/sklearn/__init__.py
@@ -17,13 +17,16 @@
"""Scikit-Learn models interfacing tools.
-Due to the variety of model classes provided by scikit-learn
-and to the way their learning process is implemented, model-
-specific interfaces are required for declearn compatibility.
+Due to the variety of model classes provided by scikit-learn and to the way
+their learning process is implemented, model-specific interfaces are required
+for declearn compatibility.
-This module currently implements:
-* NumpyVector: Vector subclass to wrap numpy.ndarray objects
-* SklearnSGDModel: interface to SGD-based linear models
+This module exposes the following classes:
+
+* [NumpyVector][declearn.model.sklearn.NumpyVector]:
+ Vector subclass to wrap numpy.ndarray objects.
+* [SklearnSGDModel][declearn.model.sklearn.SklearnSGDModel]:
+ Model subclass interfacing SGD-based linear models.
"""
from ._np_vec import NumpyVector
diff --git a/declearn/model/sklearn/_sgd.py b/declearn/model/sklearn/_sgd.py
index eabc2883ed169072bae0e31219082ec72703d1c2..c53cab85144979caabac182426cdc5aa94271d3e 100644
--- a/declearn/model/sklearn/_sgd.py
+++ b/declearn/model/sklearn/_sgd.py
@@ -179,11 +179,11 @@ class SklearnSGDModel(Model):
Notes
-----
- Save for `kind`, all parameters are strictly equivalent to those
- of `sklearn.linear_modelSGDClassifier` and `SGDRegressor`. Refer
- to the latter' documentation for additional details.
- Note that unexposed parameters from those classes are simply not
- used and/or overwritten when wrapped by `SklearnSGDModel`.
+ - Save for `kind`, all parameters are strictly equivalent to those
+ of `sklearn.linear_modelSGDClassifier` and `SGDRegressor`. Refer
+ to the latter' documentation for additional details.
+ - Note that unexposed parameters from those classes are simply not
+ used and/or overwritten when wrapped by `SklearnSGDModel`.
Returns
-------
diff --git a/declearn/model/tensorflow/__init__.py b/declearn/model/tensorflow/__init__.py
index ff94c495b4647614cc5f9e9e44b0a9d636f3c3d3..886ff3472c98c0b3a86812db88925cdfe6925fed 100644
--- a/declearn/model/tensorflow/__init__.py
+++ b/declearn/model/tensorflow/__init__.py
@@ -17,13 +17,15 @@
"""Tensorflow models interfacing tools.
-This submodule provides with a generic interface to wrap up
-any TensorFlow `keras.Model` instance that is to be trained
-through gradient descent.
+This submodule provides with a generic interface to wrap up any TensorFlow
+`tensorflow.keras.Model` instance that is to be trained with gradient descent.
-This module exposes:
-* TensorflowModel: Model subclass to wrap tensorflow.keras.Model objects
-* TensorflowVector: Vector subclass to wrap tensorflow.Tensor objects
+It exposes the following classes:
+
+* [TensorflowModel][declearn.model.tensorflow.TensorflowModel]:
+ Model subclass to wrap tensorflow.keras.Model objects.
+* [TensorflowVector][declearn.model.tensorflow.TensorflowVector]:
+ Vector subclass to wrap tensorflow.Tensor objects.
"""
from ._vector import TensorflowVector
diff --git a/declearn/model/tensorflow/_model.py b/declearn/model/tensorflow/_model.py
index fc1a9eb88dc4d42f07314276b266d20749661318..f173526cb20a55fc29d433812087b49373a636cf 100644
--- a/declearn/model/tensorflow/_model.py
+++ b/declearn/model/tensorflow/_model.py
@@ -172,7 +172,7 @@ class TensorflowModel(Model):
Raises
------
- KeyError:
+ KeyError
In case some expected keys are missing, or additional keys
are present. Be verbose about the identified mismatch(es).
"""
diff --git a/declearn/model/torch/__init__.py b/declearn/model/torch/__init__.py
index 352d3a398f6958488404c2ecd5aecc4a0f29a43b..70d51e5f49ef35d8cecf0fbce9acedb32848cd52 100644
--- a/declearn/model/torch/__init__.py
+++ b/declearn/model/torch/__init__.py
@@ -17,13 +17,15 @@
"""Tensorflow models interfacing tools.
-This submodule provides with a generic interface to wrap up
-any PyTorch `nn.Module` instance that is to be trained with
-gradient descent.
+This submodule provides with a generic interface to wrap up any PyTorch
+`torch.nn.Module` instance that is to be trained with gradient descent.
-This module exposes:
-* TorchModel: Model subclass to wrap torch.nn.Module objects
-* TorchVector: Vector subclass to wrap torch.Tensor objects
+This module exposes the following classes:
+
+* [TorchModel][declearn.model.torch.TorchModel]:
+ Model subclass to wrap torch.nn.Module objects.
+* [TorchVector][declearn.model.torch.TorchVector]:
+ Vector subclass to wrap torch.Tensor objects.
"""
from ._vector import TorchVector
diff --git a/declearn/optimizer/__init__.py b/declearn/optimizer/__init__.py
index 00018c09b9535d64c1793f13463fabefc287d958..520eeafa235b9b7ebce84c7327c96455deb09de0 100644
--- a/declearn/optimizer/__init__.py
+++ b/declearn/optimizer/__init__.py
@@ -22,13 +22,17 @@ of plug-in modules, used to implement various optimization and regularization
techniques.
Main class:
-* Optimizer: Base class to define gradient-descent-based optimizers.
-This module also implements the following submodules, used by the former:
-* modules: gradients-alteration algorithms, implemented as plug-in modules.
-* regularizers: loss-regularization algorithms, implemented as plug-in modules.
+* [Optimizer][declearn.optimizer.Optimizer]:
+ Base class to define gradient-descent-based optimizers.
- """
+Submodules providing with plug-in algorithms:
+
+* [modules][declearn.optimizer.modules]:
+ Gradients-alteration algorithms, implemented as plug-in modules.
+* [regularizers][declearn.optimizer.regularizers]:
+ Loss-regularization algorithms, implemented as plug-in modules.
+"""
from . import modules, regularizers
diff --git a/declearn/optimizer/_base.py b/declearn/optimizer/_base.py
index c3c723bfff697b52fa5720e40f66f80af4e69aa6..65e43eca7818a5c78fbf0c84599c0525dda94431 100644
--- a/declearn/optimizer/_base.py
+++ b/declearn/optimizer/_base.py
@@ -39,17 +39,19 @@ class Optimizer:
It is also fully-workable and is designed to be customizable
through the use of "plug-in modules" rather than subclassing
(which might be used for advanced algorithm modifications):
- see `declearn.optimizer.modules.OptiModule` for API details.
+ see the base classes [declearn.optimizer.modules.OptiModule][]
+ and [declearn.optimizer.regularizers.Regularizer][] for details.
The process implemented here is the following:
- * Compute or receive the (pseudo-)gradients of a model.
- * Compute loss-regularization terms and add them to the
+
+ - Compute or receive the (pseudo-)gradients of a model.
+ - Compute loss-regularization terms and add them to the
gradients, based on a list of plug-in regularizers.
- * Refine gradients by running them through plug-in modules,
+ - Refine gradients by running them through plug-in modules,
which are thus composed by sequential application.
- * Optionally compute a decoupled weight decay term (see [1])
+ - Optionally compute a decoupled weight decay term (see [1])
and add it to the updates (i.e. refined gradients).
- * Apply the learning rate and perform the weights' udpate.
+ - Apply the learning rate and perform the weights' udpate.
Most plug-in modules are self-contained, in the sense that they
do not require any information flow between the server and its
@@ -70,7 +72,6 @@ class Optimizer:
Attributes
----------
-
lrate: float
Base learning rate applied to computed updates.
w_decay: float
@@ -82,7 +83,7 @@ class Optimizer:
List of plug-in loss regularization modules composed into
the optimizer's gradients-to-updates computation algorithm.
- API methods:
+ API methods
-----------
apply_gradients(Model, Vector) -> None:
Update a Model based on a pre-computed Vector of gradients.
@@ -246,7 +247,7 @@ class Optimizer:
Raises
------
- KeyError:
+ KeyError
If the provided `config` lacks some required parameters
and/or contains some unused ones.
"""
@@ -444,13 +445,13 @@ class Optimizer:
Raises
------
- KeyError:
+ KeyError
If the received states do not match the expected config,
whether because a module is missing or one of its states
is missing.
In both cases, the Optimizer's states will be reverted
to their values prior to the failed call to this method.
- RuntimeError:
+ RuntimeError
If a KeyError was raised both when trying to apply the
input `state` and when trying to revert the states to
their initial values after that first error was raised.
diff --git a/declearn/optimizer/modules/__init__.py b/declearn/optimizer/modules/__init__.py
index d24cf4315210da8a25b1a228d812163e1c4e3471..265737ee343c000124977c62840bd84e348a095a 100644
--- a/declearn/optimizer/modules/__init__.py
+++ b/declearn/optimizer/modules/__init__.py
@@ -17,58 +17,74 @@
"""Optimizer gradients-alteration algorithms, implemented as plug-in modules.
-Base class implemented here:
-* OptiModule: base API for optimizer plug-in algorithms
+API base class
+--------------
+* [OptiModule][declearn.optimizer.modules.OptiModule]:
+ Abstact base class for optimizer plug-in algorithms.
-Adaptive learning-rate algorithms:
-* AdaGradModule : AdaGrad algorithm
-* AdamModule : Adam and AMSGrad algorithms
-* RMSPropModule : RMSProp algorithm
-* YogiModule : Yogi algorithm, with Adam or AMSGrad base
+Adaptive learning-rate algorithms
+---------------------------------
+* [AdaGradModule][declearn.optimizer.modules.AdaGradModule]:
+ AdaGrad algorithm.
+* [AdamModule][declearn.optimizer.modules.AdamModule]:
+ Adam and AMSGrad algorithms.
+* [RMSPropModule][declearn.optimizer.modules.RMSPropModule]:
+ RMSProp algorithm.
+* [YogiModule][declearn.optimizer.modules.YogiModule]:
+ Yogi algorithm, with Adam or AMSGrad base.
-Gradient clipping algorithms:
-* L2Clipping : Fixed-threshold L2-norm gradient clipping module
+Gradient clipping algorithms
+----------------------------
+* [L2Clipping][declearn.optimizer.modules.L2Clipping]:
+ Fixed-threshold L2-norm gradient clipping module.
-Momentum algorithms:
-* EWMAModule : Exponentially-Weighted Moving Average module
-* MomentumModule : Momentum (and Nesterov) acceleration module
-* YogiMomentumModule : Yogi-specific EWMA-like module
+Momentum algorithms
+-------------------
+* [EWMAModule][declearn.optimizer.modules.EWMAModule]:
+ Exponentially-Weighted Moving Average module.
+* [MomentumModule][declearn.optimizer.modules.MomentumModule]:
+ Momentum (and Nesterov) acceleration module.
+* [YogiMomentumModule][declearn.optimizer.modules.YogiMomentumModule]:
+ Yogi-specific EWMA-like module.
-Noise-addition mechanisms:
-* NoiseModule : abstract base class for noise-addition modules
-* GaussianNoiseModule : Gaussian noise-addition module
+Noise-addition mechanisms
+-------------------------
+* [NoiseModule][declearn.optimizer.modules.NoiseModule]:
+ Abstract base class for noise-addition modules.
+* [GaussianNoiseModule][declearn.optimizer.modules.GaussianNoiseModule]:
+ Gaussian noise-addition module.
-SCAFFOLD algorithm, as a pair of complementary modules:
-* ScaffoldClientModule : client-side module
-* ScaffoldServerModule : server-side module
+SCAFFOLD algorithm
+------------------
+Scaffold is implemented as a pair of complementary modules:
+
+* [ScaffoldClientModule][declearn.optimizer.modules.ScaffoldClientModule]:
+ Client-side Scaffold module.
+* [ScaffoldServerModule][declearn.optimizer.modules.ScaffoldServerModule]:
+ Server-side Scaffold module.
"""
from ._api import (
OptiModule,
)
-
from ._adaptive import (
AdaGradModule,
AdamModule,
RMSPropModule,
YogiModule,
)
-
from ._clipping import (
L2Clipping,
)
-
from ._momentum import (
EWMAModule,
MomentumModule,
YogiMomentumModule,
)
-
from ._noise import (
GaussianNoiseModule,
NoiseModule,
)
-
from ._scaffold import (
ScaffoldClientModule,
ScaffoldServerModule,
diff --git a/declearn/optimizer/modules/_adaptive.py b/declearn/optimizer/modules/_adaptive.py
index 54ff5e794e53eb98fc330b7c86174b030b003f2b..b0e06f38b07d2c603f82d2ab79de72c475055787 100644
--- a/declearn/optimizer/modules/_adaptive.py
+++ b/declearn/optimizer/modules/_adaptive.py
@@ -35,6 +35,7 @@ class AdaGradModule(OptiModule):
"""Adaptative Gradient Algorithm (AdaGrad) module.
This module implements the following algorithm:
+
Init(eps):
state = 0
Step(grads):
@@ -45,7 +46,8 @@ class AdaGradModule(OptiModule):
are scaled down by the square-root of the sum of the past
squared gradients. See reference [1].
- References:
+ References
+ ----------
[1] Duchi et al., 2012.
Adaptive Subgradient Methods for Online Learning
and Stochastic Optimization.
@@ -100,6 +102,7 @@ class RMSPropModule(OptiModule):
"""Root Mean Square Propagation (RMSProp) module.
This module implements the following algorithm:
+
Init(beta, eps):
state = 0
Step(grads, step):
@@ -110,7 +113,8 @@ class RMSPropModule(OptiModule):
are scaled down by the square-root of the momentum-corrected
sum of the past squared gradients. See reference [1].
- References:
+ References
+ ----------
[1] Tieleman and Hinton, 2012.
Lecture 6.5-rmsprop: Divide the Gradient by a Running
Average of its Recent Magnitude.
@@ -166,6 +170,7 @@ class AdamModule(OptiModule):
"""Adaptive Moment Estimation (Adam) module.
This module implements the following algorithm:
+
Init(beta_1, beta_2, eps):
state_m = 0
state_v = 0
@@ -189,11 +194,14 @@ class AdamModule(OptiModule):
at least from the point of view of this module (a warm-up
schedule might for example counteract this).
- References:
- [1] Kingma and Ba, 2014.
+ References
+ ----------
+ - [1]
+ Kingma and Ba, 2014.
Adam: A Method for Stochastic Optimization.
https://arxiv.org/abs/1412.6980
- [2] Reddi et al., 2018.
+ - [2]
+ Reddi et al., 2018.
On the Convergence of Adam and Beyond.
https://arxiv.org/abs/1904.09237
"""
@@ -289,6 +297,7 @@ class YogiModule(AdamModule):
"""Yogi additive adaptive moment estimation module.
This module implements the following algorithm:
+
Init(beta_1, beta_2, eps):
state_m = 0
state_v = 0
@@ -307,13 +316,17 @@ class YogiModule(AdamModule):
Note that this implementation allows combining the Yogi
modification of Adam with the AMSGrad [3] one.
- References:
- [1] Zaheer and Reddi et al., 2018.
+ References
+ ----------
+ - [1]
+ Zaheer and Reddi et al., 2018.
Adaptive Methods for Nonconvex Optimization.
- [2] Kingma and Ba, 2014.
+ - [2]
+ Kingma and Ba, 2014.
Adam: A Method for Stochastic Optimization.
https://arxiv.org/abs/1412.6980
- [3] Reddi et al., 2018.
+ - [3]
+ Reddi et al., 2018.
On the Convergence of Adam and Beyond.
https://arxiv.org/abs/1904.09237
"""
diff --git a/declearn/optimizer/modules/_api.py b/declearn/optimizer/modules/_api.py
index bc0ba4460ae336701d054e6b625c8c95f3c71220..197346e0bc425f94e053cf336d3eb05bfd5e5468 100644
--- a/declearn/optimizer/modules/_api.py
+++ b/declearn/optimizer/modules/_api.py
@@ -56,11 +56,11 @@ class OptiModule(metaclass=ABCMeta):
The following attribute and method require to be overridden
by any non-abstract child class of `OptiModule`:
- name: str class attribute
+ - name: str class attribute
Name identifier of the class (should be unique across existing
OptiModule classes). Also used for automatic types-registration
of the class (see `Inheritance` section below).
- run(gradients: Vector) -> Vector:
+ - run(gradients: Vector) -> Vector:
Apply an adaptation algorithm to input gradients and return
them. This is the main method for any `OptiModule`.
@@ -71,15 +71,15 @@ class OptiModule(metaclass=ABCMeta):
As defined at `OptiModule` level, they have no effect and may thus
be safely ignored when implementing self-contained algorithms.
- collect_aux_var() -> Optional[Dict[str, Any]]:
+ - collect_aux_var() -> Optional[Dict[str, Any]]:
Emit a JSON-serializable dict of auxiliary variables,
to be received by a counterpart of this module on the
other side of the client/server relationship.
- process_aux_var(Dict[str, Any]) -> None:
+ - process_aux_var(Dict[str, Any]) -> None:
Process a dict of auxiliary variables, received from
a counterpart to this module on the other side of the
client/server relationship.
- aux_name: optional[str] class attribute, default=None
+ - aux_name: optional[str] class attribute, default=None
Name to use when sending or receiving auxiliary variables
between synchronous client/server modules, that therefore
need to share the *same* `aux_name`.
@@ -94,7 +94,15 @@ class OptiModule(metaclass=ABCMeta):
"""
name: ClassVar[str] = NotImplemented
+ """Name identifier of the class, unique across OptiModule classes."""
+
aux_name: ClassVar[Optional[str]] = None
+ """Optional aux-var-sharing identifier of the class.
+
+ This name may be shared by a pair of OptiModule classes, designed
+ to operate on the client and server side respectively. It should
+ be unique to that pair of classes across all OptiModule classes.
+ """
def __init_subclass__(
cls,
@@ -137,7 +145,7 @@ class OptiModule(metaclass=ABCMeta):
Returns
-------
- aux_var: dict[str, any] or None
+ aux_var: Optional[Dict[str, Any]]
Optional JSON-serializable dict of auxiliary variables that
are to be shared with a similarly-named OptiModule on the
other side of the client-server relationship.
@@ -146,20 +154,21 @@ class OptiModule(metaclass=ABCMeta):
-----
Specfications for the output and calling context depend on whether
the module is part of a client's optimizer or of the server's one:
- * Client:
- - aux_var is dict[str, any] or None.
- - `collect_aux_var` is expected to happen after taking a series
- of local optimization steps, before sending the local updates
- to the server for aggregation and further processing.
- * Server:
- - aux_var may be None ; dict[str, any] (to send the same values
- to each and every client) ; or dict[str, dict[str, any]] with
- clients' names as keys and client-wise new aux_var as values
- so as to send distinct values to the clients.
- - `collect_aux_var` is expected to happen when the global model
- weights are ready to be shared with clients, i.e. at the very
- end of a training round or at the beginning of the training
- process.
+
+ - Client:
+ - `aux_var` is dict[str, any] or None.
+ - `collect_aux_var` is expected to happen after taking a series
+ of local optimization steps, before sending the local updates
+ to the server for aggregation and further processing.
+ - Server:
+ - `aux_var` may be None ; dict[str, any] (to send the same values
+ to each and every client) ; or dict[str, dict[str, any]] with
+ clients' names as keys and client-wise new aux_var as values
+ so as to send distinct values to the clients.
+ - `collect_aux_var` is expected to happen when the global model
+ weights are ready to be shared with clients, i.e. at the very
+ end of a training round or at the beginning of the training
+ process.
"""
return None
@@ -181,24 +190,25 @@ class OptiModule(metaclass=ABCMeta):
-----
Specfications for the inputs and calling context depend on whether
the module is part of a client's optimizer or of the server's one:
- * Client:
- - aux_var is dict[str, any] and may be client-specific.
- - `process_aux_var` is expected to happen at the beginning of
- a training round to define gradients' processing during the
- local optimization steps taken through that round.
- * Server:
- - aux_var is dict[str, dict[str, any]] with clients' names as
- primary keys and client-wise collected aux_var as values.
- - `process_aux_var` is expected to happen upon receiving local
- updates (and, thus, aux_var), before the aggregated updates
- are computed and passed through the server optimizer (which
- comprises this module).
+
+ - Client:
+ - `aux_var` is dict[str, any] and may be client-specific.
+ - `process_aux_var` is expected to happen at the beginning of
+ a training round to define gradients' processing during the
+ local optimization steps taken through that round.
+ - Server:
+ - `aux_var` is dict[str, dict[str, any]] with clients' names as
+ primary keys and client-wise collected aux_var as values.
+ - `process_aux_var` is expected to happen upon receiving local
+ updates (and, thus, aux_var), before the aggregated updates
+ are computed and passed through the server optimizer (which
+ comprises this module).
Raises
------
- KeyError:
+ KeyError
If an expected auxiliary variable is missing.
- TypeError:
+ TypeError
If a variable is of unproper type, or if aux_var
is not formatted as it should be.
"""
@@ -214,7 +224,7 @@ class OptiModule(metaclass=ABCMeta):
Returns
-------
- config: dict[str, any]
+ config: Dict[str, Any]
JSON-serializable dict storing this module's instantiation
configuration.
"""
@@ -238,7 +248,7 @@ class OptiModule(metaclass=ABCMeta):
Raises
------
- KeyError:
+ KeyError
If the provided `config` lacks some required parameters
and/or contains some unused ones.
"""
@@ -273,7 +283,7 @@ class OptiModule(metaclass=ABCMeta):
Returns
-------
- state: dict[str, any]
+ state: Dict[str, Any]
JSON-serializable dict storing this module's inner state
variables.
"""
@@ -295,7 +305,7 @@ class OptiModule(metaclass=ABCMeta):
Raises
------
- KeyError:
+ KeyError
If an expected state variable is missing from `state`.
"""
# API-defining method; pylint: disable=unused-argument
diff --git a/declearn/optimizer/modules/_clipping.py b/declearn/optimizer/modules/_clipping.py
index 24756b2bf632fe76b82fcbe0b7b708a83fc92973..5343950d43c505af8938ccf4b151e21ca42c8f17 100644
--- a/declearn/optimizer/modules/_clipping.py
+++ b/declearn/optimizer/modules/_clipping.py
@@ -29,6 +29,7 @@ class L2Clipping(OptiModule):
"""Fixed-threshold L2-norm gradient clipping module.
This module implements the following algorithm:
+
Init(max_norm):
Step(max_norm):
norm = euclidean_norm(grads)
diff --git a/declearn/optimizer/modules/_momentum.py b/declearn/optimizer/modules/_momentum.py
index 0923ceb0baed5b5b4865e31d33e7f0d399946c73..a9c2808ecc94b5ed8d7aa803144ee70cb5a6b3ac 100644
--- a/declearn/optimizer/modules/_momentum.py
+++ b/declearn/optimizer/modules/_momentum.py
@@ -33,6 +33,7 @@ class MomentumModule(OptiModule):
"""Momentum gradient-acceleration module.
This module impements the following algorithm:
+
Init(beta):
velocity = 0
Step(grads):
@@ -45,16 +46,18 @@ class MomentumModule(OptiModule):
Note that this contrasts with the canonical implementation of momentum
by Sutskever et. al. [1]. The learning rate is applied to the whole output
of the algorithm above, in the Optmizer class, rather than only to the
- gradient part of it, following the [pytorch implementation]\
- (https://pytorch.org/docs/stable/generated/torch.optim.SGD.html).
+ gradient part of it, following the [pytorch implementation](\
+ https://pytorch.org/docs/stable/generated/torch.optim.SGD.html).
The nesterov variant's implementation is equivalently adapted.
- This formaluation is equivalent to the canonical one for constant learning
+ This formulation is equivalent to the canonical one for constant learning
rare (eta), with both approaches outputting:
- $$ w_{t+1} = w_t - \\eta \\sum_{k=1}^t \\beta^{t-k} \nabla_k $$
+ $$ w_{t+1} = w_t - \\eta \\sum_{k=1}^t \\beta^{t-k} \\nabla_k $$
+
It may however yield differences when $\\eta$ changes through training:
- (can.) $$ w_{t+1} = w_t - \\sum_{k=1}^t \\eta_k \\beta^{t-k} \\nabla_k $$
- (ours) $$ w_{t+1} = w_t - \\eta_t \\sum_{k=1}^t \\beta^{t-k} \\nabla_k $$
+
+ - (can.) $$ w_{t+1} = w_t - \\sum_{k=1}^t \\eta_k \\beta^{t-k} \\nabla_k $$
+ - (ours) $$ w_{t+1} = w_t - \\eta_t \\sum_{k=1}^t \\beta^{t-k} \\nabla_k $$
References
----------
@@ -119,6 +122,7 @@ class EWMAModule(OptiModule):
"""Exponentially Weighted Moving Average module.
This module impements the following algorithm:
+
Init(beta):
state = 0
Step(grads):
@@ -180,6 +184,7 @@ class YogiMomentumModule(EWMAModule):
"""Yogi-specific momentum gradient-acceleration module.
This module impements the following algorithm:
+
Init(beta):
state = 0
Step(grads):
@@ -194,7 +199,8 @@ class YogiMomentumModule(EWMAModule):
Note that this module is actually meant to be used to compute
a learning-rate adaptation term based on squared gradients.
- References:
+ References
+ ----------
[1] Zaheer and Reddi et al., 2018.
Adaptive Methods for Nonconvex Optimization.
"""
diff --git a/declearn/optimizer/modules/_scaffold.py b/declearn/optimizer/modules/_scaffold.py
index 46cae6b4c6affccdd690b02da81c3d982f6445c6..1d0f0280f7575a1603c08cd906064f3c73a5f5c3 100644
--- a/declearn/optimizer/modules/_scaffold.py
+++ b/declearn/optimizer/modules/_scaffold.py
@@ -25,7 +25,8 @@ They only implement Option-II of the paper regarding client-specific
state variables' update, and implementing Option-I would require the
use of a specific Optimizer sub-class.
-References:
+References
+----------
[1] Karimireddy et al., 2019.
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning.
https://arxiv.org/abs/1910.06378
@@ -47,9 +48,11 @@ class ScaffoldClientModule(OptiModule):
This module is to be added to the optimizer used by a federated-
learning client, and expects that the server's optimizer use its
- counterpart module: `ScaffoldServerModule`.
+ counterpart module:
+ [`ScaffoldServerModule`][declearn.optimizer.modules.ScaffoldServerModule].
This module implements the following algorithm:
+
Init:
delta = 0
_past = 0
@@ -87,7 +90,8 @@ class ScaffoldClientModule(OptiModule):
in `declearn`, but requires implementing an alternative training
procedure rather than an optimizer plug-in.
- References:
+ References
+ ----------
[1] Karimireddy et al., 2019.
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning.
https://arxiv.org/abs/1910.06378
@@ -116,12 +120,26 @@ class ScaffoldClientModule(OptiModule):
def collect_aux_var(
self,
- ) -> Optional[Dict[str, Any]]:
+ ) -> Dict[str, Any]:
"""Return auxiliary variables that need to be shared between nodes.
Compute and package (without applying it) the updated value
of the local state variable, so that the server may compute
the updated shared state variable.
+
+ Returns
+ -------
+ aux_var: dict[str, any]
+ JSON-serializable dict of auxiliary variables that
+ are to be shared with a ScaffoldServerModule held
+ by the orchestrating server.
+
+ Raises
+ ------
+ RuntimeError
+ If called on an instance that has not processed any gradients
+ (via a call to `run`) since the last call to `process_aux_var`
+ (or its instantiation).
"""
state = self._compute_updated_state()
return {"state": state}
@@ -165,6 +183,22 @@ class ScaffoldClientModule(OptiModule):
Collect the (local_state - shared_state) variable sent by server.
Reset hidden variables used to compute the local state's updates.
+
+ Parameters
+ ----------
+ aux_var: dict[str, any]
+ JSON-serializable dict of auxiliary variables that are to be
+ processed by this module at the start of a training round (on
+ the client side).
+ Expected keys for this class: {"delta"}.
+
+ Raises
+ ------
+ KeyError
+ If an expected auxiliary variable is missing.
+ TypeError
+ If a variable is of unproper type, or if aux_var
+ is not formatted as it should be.
"""
# Expect a state variable and apply it.
delta = aux_var.get("delta", None)
@@ -190,9 +224,11 @@ class ScaffoldServerModule(OptiModule):
This module is to be added to the optimizer used by a federated-
learning server, and expects that the clients' optimizer use its
- counterpart module: `ScaffoldClientModule`.
+ counterpart module:
+ [`ScaffoldClientModule`][declearn.optimizer.modules.ScaffoldClientModule].
This module implements the following algorithm:
+
Init(clients):
state = 0
s_loc = {client: 0 for client in clients}
@@ -221,7 +257,8 @@ class ScaffoldServerModule(OptiModule):
The client-side correction of gradients and the computation of
updated local states are deferred to `ScaffoldClientModule`.
- References:
+ References
+ ----------
[1] Karimireddy et al., 2019.
SCAFFOLD: Stochastic Controlled Averaging for Federated Learning.
https://arxiv.org/abs/1910.06378
@@ -241,14 +278,16 @@ class ScaffoldServerModule(OptiModule):
clients: list[str] or None, default=None
Optional list of known clients' id strings.
- If this module is used under a training strategy that has
- participating clients vary across epochs, leaving `clients`
- to None will affect the update rule for the shared state,
- as it uses a (n_participating / n_total_clients) term, the
- divisor of which will be incorrect (at least on the first
- step, potentially on following ones as well).
- Similarly, listing clients that in fact do not participate
- in training will have side effects on computations.
+ Notes
+ -----
+ - If this module is used under a training strategy that has
+ participating clients vary across epochs, leaving `clients`
+ to None will affect the update rule for the shared state,
+ as it uses a (n_participating / n_total_clients) term, the
+ divisor of which will be incorrect (at least on the first
+ step, potentially on following ones as well).
+ - Similarly, listing clients that in fact do not participate
+ in training will have side effects on computations.
"""
self.state = 0.0 # type: Union[Vector, float]
self.s_loc = {} # type: Dict[str, Union[Vector, float]]
@@ -269,10 +308,17 @@ class ScaffoldServerModule(OptiModule):
def collect_aux_var(
self,
- ) -> Optional[Dict[str, Any]]:
+ ) -> Dict[str, Dict[str, Any]]:
"""Return auxiliary variables that need to be shared between nodes.
- Package client-wise (local_state - shared_state) variables.
+ Package client-wise `delta = (local_state - shared_state)` variables.
+
+ Returns
+ -------
+ aux_var:
+ JSON-serializable dict of auxiliary variables that are to
+ be shared with the client-wise ScaffoldClientModule. This
+ dict has a `{client-name: {"delta": value}}` structure.
"""
# Compute clients' delta variable, package them and return.
aux_var = {} # type: Dict[str, Dict[str, Any]]
@@ -283,12 +329,28 @@ class ScaffoldServerModule(OptiModule):
def process_aux_var(
self,
- aux_var: Dict[str, Any],
+ aux_var: Dict[str, Dict[str, Any]],
) -> None:
"""Update this module based on received shared auxiliary variables.
Collect updated local state variables sent by clients.
Update the global state variable based on the latter.
+
+ Parameters
+ ----------
+ aux_var: dict[str, dict[str, any]]
+ JSON-serializable dict of auxiliary variables that are to be
+ processed by this module before processing global updates.
+ This dict should have a `{client-name: {"state": value}}`
+ structure.
+
+ Raises
+ ------
+ KeyError:
+ If an expected auxiliary variable is missing.
+ TypeError:
+ If a variable is of unproper type, or if aux_var
+ is not formatted as it should be.
"""
# Collect updated local states received from Scaffold client modules.
s_new = {} # type: Dict[str, Union[Vector, float]]
diff --git a/declearn/optimizer/regularizers/__init__.py b/declearn/optimizer/regularizers/__init__.py
index 87c797080dec30547842e769415092ff9c95f10b..0f6ead5146b28d07d4952cdcac92add819e7c1e4 100644
--- a/declearn/optimizer/regularizers/__init__.py
+++ b/declearn/optimizer/regularizers/__init__.py
@@ -17,15 +17,21 @@
"""Optimizer loss-regularization algorithms, implemented as plug-in modules.
-Base class implemented here:
+API base class
+--------------
* Regularizer: base API for loss-regularization plug-ins
-Common regularization terms:
-* LassoRegularizer : L1 regularization, aka Lasso penalization
-* RidgeRegularizer : L2 regularization, aka Ridge penalization
+Common regularization terms
+---------------------------
+* [LassoRegularizer][declearn.optimizer.regularizers.LassoRegularizer]:
+ L1 regularization, aka Lasso penalization.
+* [RidgeRegularizer][declearn.optimizer.regularizers.RidgeRegularizer]:
+ L2 regularization, aka Ridge penalization.
-Federated-Learning-specific regularizers:
-* FedProxRegularizer : FedProx algorithm, as a proximal term regularizer
+Federated-Learning-specific regularizers
+----------------------------------------
+* [FedProxRegularizer][declearn.optimizer.regularizers.FedProxRegularizer]:
+ FedProx algorithm, as a proximal term regularizer.
"""
from ._api import Regularizer
diff --git a/declearn/optimizer/regularizers/_api.py b/declearn/optimizer/regularizers/_api.py
index 2c96f854b2ce4c50da311ea65906238d0fd18f82..1a9dfa5bc2151971a101c48e11ce7c480d4ec7ca 100644
--- a/declearn/optimizer/regularizers/_api.py
+++ b/declearn/optimizer/regularizers/_api.py
@@ -40,9 +40,10 @@ class Regularizer(metaclass=ABCMeta):
The `Regularizer` API is close to the `OptiModule` one, with the
following differences:
- * Regularizers are meant to be applied prior to Modules, as a way
+
+ - Regularizers are meant to be applied prior to Modules, as a way
to complete the computation of "raw" gradients.
- * Regularizers do not provide an API to share stateful variables
+ - Regularizers do not provide an API to share stateful variables
between a server and its clients.
The aim of this abstraction (which itself operates on the Vector
@@ -59,18 +60,18 @@ class Regularizer(metaclass=ABCMeta):
Abstract
--------
- name: str class attribute
+ - name: str class attribute
Name identifier of the class (should be unique across existing
Regularizer classes). Also used for automatic types-registration
of the class (see `Inheritance` section below).
- run(gradients: Vector, weights: Vector) -> Vector:
+ - run(gradients: Vector, weights: Vector) -> Vector:
Compute the regularization term's derivative from weights,
and add it to the input gradients. This is the main method
for any `Regularizer`.
Overridable
-----------
- on_round_start() -> None:
+ - on_round_start() -> None:
Perform any required operation (e.g. resetting a state variable)
at the start of a training round. By default, this method has no
effect and mey thus be safely ignored when no behavior is needed.
@@ -85,6 +86,7 @@ class Regularizer(metaclass=ABCMeta):
"""
name: ClassVar[str] = NotImplemented
+ """Name identifier of the class, unique across Regularizer classes."""
def __init_subclass__(
cls,
diff --git a/declearn/optimizer/regularizers/_base.py b/declearn/optimizer/regularizers/_base.py
index f96e6f3681f217c5733a9f87c75b8eff74b01d33..a2dd97d532a678118a2db9afd55a0be7191ccb50 100644
--- a/declearn/optimizer/regularizers/_base.py
+++ b/declearn/optimizer/regularizers/_base.py
@@ -38,17 +38,20 @@ class FedProxRegularizer(Regularizer):
See paper [1].
This regularizer implements the following term:
+
loss += alpha / 2 * (weights - ref_wgt)^2
w/ ref_wgt := weights at the 1st step of the round
To do so, it applies the following correction to gradients:
+
grads += alpha * (weights - ref_wgt)
In other words, this regularizer penalizes weights' departure
(as a result from local optimization steps) from their initial
(shared) values.
- References:
+ References
+ ----------
[1] Li et al., 2020.
Federated Optimization in Heterogeneous Networks.
https://arxiv.org/abs/1812.06127
@@ -84,9 +87,11 @@ class LassoRegularizer(Regularizer):
"""L1 (Lasso) loss-regularization plug-in.
This regularizer implements the following term:
+
loss += alpha * l1_norm(weights)
To do so, it applies the following correction to gradients:
+
grads += alpha * sign(weights)
"""
@@ -105,9 +110,11 @@ class RidgeRegularizer(Regularizer):
"""L2 (Ridge) loss-regularization plug-in.
This regularizer implements the following term:
+
loss += alpha * l2_norm(weights)
To do so, it applies the following correction to gradients:
+
grads += alpha * 2 * weights
"""
diff --git a/declearn/test_utils/__init__.py b/declearn/test_utils/__init__.py
index 507d8ebffeea17c1601e4bfca51eaa2d2298adf3..a74530844ae5ba90e3c0378ec046bd1d321031b6 100644
--- a/declearn/test_utils/__init__.py
+++ b/declearn/test_utils/__init__.py
@@ -15,7 +15,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-"""Collection of utils for running tests and examples around declearn."""
+"""Collection of utils for running tests and examples around declearn.
+
+This submodule is not imported with declearn by default - it requires
+being explicitly imported, and should not be so by end-users, unless
+they accept the risk of using unstable features.
+
+This submodule is *not* considered part of the stable declearn API,
+meaning that its contents may change without warnings. Its features
+are not designed to be used outside of the scope of declearn-shipped
+tests and examples. It may also serve to introduce experimental new
+features that may be ported to the stable API in the future.
+"""
from ._assertions import assert_json_serializable_dict
from ._gen_ssl import generate_ssl_certificates
diff --git a/declearn/test_utils/_assertions.py b/declearn/test_utils/_assertions.py
index 456720c9214b8c520a07f6cf7f08df3926e373a3..776dbbee40732b1de778cbf8460f7a94eeb0523f 100644
--- a/declearn/test_utils/_assertions.py
+++ b/declearn/test_utils/_assertions.py
@@ -45,11 +45,11 @@ def assert_json_serializable_dict(sdict: Dict[str, Any]) -> None:
Raises
------
- AssertionError:
+ AssertionError
If `sdict` or the JSON-reloaded object is not a dict, or
if the latter has different keys and/or values compared
to the former.
- Exception:
+ Exception
Other exceptions may be raised if the JSON encoding (or
decoding) operation goes wrong.
"""
diff --git a/declearn/typing.py b/declearn/typing.py
index 2783c499e4dc7c1fc2e888fe8e13e4b40f0a6118..c7b1c6429a093c14fc6110de2c8acbe220668cc8 100644
--- a/declearn/typing.py
+++ b/declearn/typing.py
@@ -29,14 +29,19 @@ __all__ = [
"SupportsConfig",
]
-# Data batches specification: (inputs, labels, weights), where:
-# - inputs and labels may be an array or a list of arrays:
-# - labels and/or weights may ne None
+
Batch = Tuple[
Union[ArrayLike, List[ArrayLike]],
Optional[Union[ArrayLike, List[ArrayLike]]],
Optional[ArrayLike],
]
+"""Data batches specification type-annotation.
+
+This type-hint designates (inputs, labels, weights) inputs, where:
+
+- inputs and labels may be an array or a list of arrays;
+- labels and/or weights may be None;
+""" # this is rendered as a docstring for `Batch` in the docs
class SupportsConfig(Protocol, metaclass=ABCMeta):
diff --git a/declearn/utils/__init__.py b/declearn/utils/__init__.py
index 00bf9fc14932fef61b6eaa4ab9b8aefeb404374c..228511396125b20a2acd41587902d6fd3b956abd 100644
--- a/declearn/utils/__init__.py
+++ b/declearn/utils/__init__.py
@@ -17,17 +17,18 @@
"""Shared utils used across declearn.
-The key functionalities implemented here are:
+The functions and classes exposed by this submodule are listed below,
+grouped thematically.
Config serialization
--------------------
Tools to create JSON config dumps of objects and instantiate from them.
-* ObjectConfig:
+* [ObjectConfig][declearn.utils.ObjectConfig]:
Dataclass to wrap objects' config and interface JSON dumps.
-* deserialize_object:
+* [deserialize_object][declearn.utils.deserialize_object]:
Instantiate an object from an ObjectConfig or a JSON file.
-* serialize_object:
+* [serialize_object][declearn.utils.serialize_object]:
Return an ObjectConfig wrapping a given (supported) object.
@@ -35,15 +36,15 @@ Types-registration
------------------
Tools to map class constructors to (name, group) string tuples.
-* access_registered:
+* [access_registered][declearn.utils.access_registered]:
Retrieve a registered type from its name and (opt.) group name.
-* access_registration_info:
+* [access_registration_info][declearn.utils.access_registration_info]:
Retrieve the name (and opt. group) under which a type is registered.
-* access_types_mapping:
+* [access_types_mapping][declearn.utils.access_types_mapping]:
Return a copy of the `{name: type}` mapping of a given group.
-* create_types_registry:
+* [create_types_registry][declearn.utils.create_types_registry]:
Create a types group from a base class (as a function or class-decorator).
-* register_type:
+* [register_type][declearn.utils.register_type]:
Register a type class (as a function or class-decorator).
@@ -51,32 +52,33 @@ JSON-serialization
------------------
Tools to add support for 3rd-party or custom types in JSON files.
-* add_json_support:
+* [add_json_support][declearn.utils.add_json_support]:
Register a (pack, unpack) pair of functions to use on a given type.
-* json_dump:
+* [json_dump][declearn.utils.json_dump]:
Function to dump data to a JSON file, automatically using `json_pack`.
-* json_load:
+* [json_load][declearn.utils.json_load]:
Function to load data from a JSON file, automatically using `json_unpack`.
-* json_pack:
+* [json_pack][declearn.utils.json_pack]:
Function to use as `default` parameter in `json.dump` to extend it.
-* json_unpack:
+* [json_unpack][declearn.utils.json_unpack]:
Function to use as `object_hook` parameter in `json.load` to extend it.
And examples of pre-registered (de)serialization functions:
-* (deserialize_numpy, serialize_numpy):
+* [deserialize_numpy][declearn.utils.deserialize_numpy]
+ and [serialize_numpy][declearn.utils.serialize_numpy]:
Pair of functions to (un)pack a numpy ndarray as JSON-serializable data.
Miscellaneous
-------------
-* TomlConfig:
+* [TomlConfig][declearn.utils.TomlConfig]:
Abstract base class to define TOML-parsable configuration containers.
-* dataclass_from_func:
+* [dataclass_from_func][declearn.utils.dataclass_from_func]:
Automatically build a dataclass matching a function's signature.
-* dataclass_from_init:
+* [dataclass_from_init][declearn.utils.dataclass_from_init]:
Automatically build a dataclass matching a class's init signature.
-* get_logger:
+* [get_logger][declearn.utils.get_logger]:
Access or create a logger, automating basic handlers' configuration.
"""
diff --git a/declearn/utils/_json.py b/declearn/utils/_json.py
index 2d6ac181e9f996ec59ea1f003fa01ba64002db12..295c21414e1e8a2828a3c52a3b771b5392c90a77 100644
--- a/declearn/utils/_json.py
+++ b/declearn/utils/_json.py
@@ -158,8 +158,10 @@ def json_dump(
"""Dump a given object to a JSON file, using extended types support.
This function is merely a shortcut to run the following code:
+ ```
>>> with open(path, "w", encoding=encoding) as file:
>>> json.dump(obj, file, default=declearn.utils.json_pack)
+ ```
See `declearn.utils.add_json_support` to extend the behaviour
of JSON (de)serialization to non-standard types, that will be
@@ -178,8 +180,10 @@ def json_load(
"""Load data from a JSON file, using extended types support.
This function is merely a shortcut to run the following code:
+ ```
>>> with open(path, "r", encoding=encoding) as file:
>>> return json.load(file, object_hook=declearn.utils.json_unpack)
+ ```
See `declearn.utils.add_json_support` to extend the behaviour
of JSON (de)serialization to non-standard types, that will be
diff --git a/declearn/utils/_register.py b/declearn/utils/_register.py
index 7453f4747171117a8e69f9ef6c68eeef2b9458a5..97935a90c21bfad70ac9f167da9fbd5db22d601d 100644
--- a/declearn/utils/_register.py
+++ b/declearn/utils/_register.py
@@ -248,7 +248,7 @@ def access_registered(
Raises
------
- KeyError:
+ KeyError
If no registered type matching the input parameters is found.
"""
# If group is unspecified, look the name up in each and every registry.
@@ -289,7 +289,7 @@ def access_registration_info(
Raises
------
- KeyError:
+ KeyError
If the provided information does not match a registered type.
"""
# If group is unspecified, look the type up in each and every registry.
@@ -326,7 +326,7 @@ def access_types_mapping(
Raises
------
- KeyError:
+ KeyError
If the `group` types registry does not exist.
"""
if group not in REGISTRIES:
diff --git a/declearn/utils/_serialize.py b/declearn/utils/_serialize.py
index 55ae5e565462382ee6ed86dcf22dd14b4a616f14..fc7d317940e2cd3b6e4a89d2af0149b53138d142 100644
--- a/declearn/utils/_serialize.py
+++ b/declearn/utils/_serialize.py
@@ -92,8 +92,8 @@ def serialize_object(
----------
obj: object
Object that needs serialization. To be valid, the object must:
- * implement the `get_config` and `from_config` (class)methods
- * belong to a registered type, unless `allow_unregistered=True`
+ - implement the `get_config` and `from_config` (class)methods
+ - belong to a registered type, unless `allow_unregistered=True`
(i.e. a type that has been passed to or decorated with the
`declearn.utils.register_type` function)
group: str or None, default=None
diff --git a/declearn/utils/_toml_config.py b/declearn/utils/_toml_config.py
index 39da17d201706cf4a075898ba4a12430e0443bc9..cfe96c2bb21b790db7eddab3f8175f9c21d2656e 100644
--- a/declearn/utils/_toml_config.py
+++ b/declearn/utils/_toml_config.py
@@ -67,7 +67,7 @@ def _isinstance_generic(inputs: Any, typevar: Type) -> bool:
Raises
------
- TypeError:
+ TypeError
If an unsupported `typevar` is provided.
"""
origin = typing.get_origin(typevar)
@@ -174,9 +174,10 @@ class TomlConfig:
The input keyword arguments should match this class's fields' names.
For each and every dataclass field of this class:
- - If unprovided, set the argument to None.
- - If a `parse_{field.name}` method exists, use that method.
- - Else, use the `default_parser` method.
+
+ - If unprovided, set the argument to None.
+ - If a `parse_{field.name}` method exists, use that method.
+ - Else, use the `default_parser` method.
Notes
-----
@@ -193,13 +194,13 @@ class TomlConfig:
Raises
------
- RuntimeError:
+ RuntimeError
In case a field failed to be instantiated using the input key-
word argument (or None value resulting from the lack thereof).
Warns
-----
- UserWarning:
+ UserWarning
In case some keyword arguments are unused due to the lack of a
corresponding dataclass field.
"""
@@ -244,13 +245,14 @@ class TomlConfig:
-----
If `inputs` is str and treated as the path to a TOML file,
it will be parsed in one of the following ways:
+
- Call `field.type.from_toml` if `field.type` is a TomlConfig.
- Use the file's `field.name` section as kwargs, if it exists.
- Use the entire file's contents as kwargs otherwise.
Raises
------
- TypeError:
+ TypeError
If instantiation failed, for any reason.
Returns
@@ -316,14 +318,14 @@ class TomlConfig:
Raises
------
- RuntimeError:
+ RuntimeError
If parsing fails, whether due to misformatting of the TOML
file, presence of undue parameters, or absence of required
ones.
Warns
-----
- UserWarning:
+ UserWarning
In case some sections of the TOML file are unused due to the
lack of a corresponding dataclass field.
"""
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
new file mode 100644
index 0000000000000000000000000000000000000000..e7d6f4d809c52ac8e981453fde74e52803c27fb8
--- /dev/null
+++ b/docs/SUMMARY.md
@@ -0,0 +1,6 @@
+- [Overview](index.md)
+- [Installation Guide](setup.md)
+- [Quickstart](quickstart.md)
+- [User Guide](user-guide/)
+- [API Reference](api-reference/)
+- [Developer Guide](devs-guide/)
diff --git a/docs/api-reference/index.md b/docs/api-reference/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..c3c54675a91122a4eb7496054cc0694415bc9b39
--- /dev/null
+++ b/docs/api-reference/index.md
@@ -0,0 +1,7 @@
+The API reference is automatically built from the source code and uploaded
+to our [website](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/api).
+This file is just a placeholder for the docs that are included as part of
+the project's main gitlab repository.
+
+You may build this doc locally: please refer to the dedicated section of
+the developers' guide on [building the docs](../devs-guide/docs-build.md).
diff --git a/docs/devs-guide/SUMMARY.md b/docs/devs-guide/SUMMARY.md
new file mode 100644
index 0000000000000000000000000000000000000000..5e63638b8fd4f0b93988a71fa84b81f059ec272e
--- /dev/null
+++ b/docs/devs-guide/SUMMARY.md
@@ -0,0 +1,5 @@
+- [Introduction](./index.md)
+- [Contributions guide](./contribute.md)
+- [Unit tests and code analysis](./tests.md)
+- [Building the documentation](./docs-build.md)
+- [Docstrings style guide](./docs-style.md)
diff --git a/docs/devs-guide/contribute.md b/docs/devs-guide/contribute.md
new file mode 100644
index 0000000000000000000000000000000000000000..7d3176300acc6314b8dbbee3e3e79691fbbcaa81
--- /dev/null
+++ b/docs/devs-guide/contribute.md
@@ -0,0 +1,65 @@
+# Contributions guide
+
+Contributions to `declearn` are welcome, whether to provide fixes, suggest
+new features (_e.g._ new subclasses of the core abstractions) or even push
+forward framework evolutions and API revisions.
+
+To contribute directly to the code (beyond posting issues on gitlab), please
+create a dedicated branch, and submit a **Merge Request** once you want your
+work reviewed and further processed to end up integrated into the package.
+
+## Git branching strategy
+
+- The 'develop' branch is the main one and should receive all finalized changes
+ to the source code. Release branches are then created and updated by cherry-
+ picking from that branch. It therefore acts as a nightly stable version.
+- The 'rX.Y' branches are release branches for each and every X.Y versions.
+ For past versions, these branches enable pushing patches towards a subminor
+ version release (hence being version `X.Y.(Z+1)-dev`). For future versions,
+ these branches enable cherry-picking commits from main to build up an alpha,
+ beta, release-candidate and eventually stable `X.Y.0` version to release.
+- Feature branches should be created at will to develop features, enhancements,
+ or even hotfixes that will later be merged into 'main' and eventually into
+ one or multiple release branches.
+- It is legit to write up poc branches, as well as to split the development of
+ a feature into multiple branches that will incrementally be merged into an
+ intermediate feature branch that will eventually be merged into 'main'.
+
+## Coding rules
+
+The **coding rules** are fairly simple:
+
+- Abide by [PEP 8](https://peps.python.org/pep-0008/), in a way that is
+ coherent with the practices already at work in declearn.
+- Abide by [PEP 257](https://peps.python.org/pep-0257/), _i.e._ write
+ docstrings **everywhere** (unless inheriting from a method, the behaviour
+ and signature of which are unmodified). The formatting rules for docstrings
+ are detailed in the [docstrings style guide](./docs-style.md).
+- Type-hint the code, abiding by [PEP 484](https://peps.python.org/pep-0484/);
+ note that the use of Any and of "type: ignore" comments is authorized, but
+ should be remain sparse.
+- Lint your code with [mypy](http://mypy-lang.org/) (for static type checking)
+ and [pylint](https://pylint.pycqa.org/en/latest/) (for more general linting);
+ do use "type: ..." and "pylint: disable=..." comments where you think it
+ relevant, preferably with some side explanations. (see dedicated sections:
+ [pylint](./tests.md#running-pylint-to-check-the-code)
+ and [mypy](./tests.md/#running-mypy-to-type-check-the-code))
+- Reformat your code using [black](https://github.com/psf/black); do use
+ (sparingly) "fmt: off/on" comments when you think it relevant (see dedicated
+ section: [black](./tests.md/#running-black-to-format-the-code)).
+- Abide by [semver](https://semver.org/) when implementing new features or
+ changing the existing APIs; try making changes non-breaking, document and
+ warn about deprecations or behavior changes, or make a point for API-breaking
+ changes, which we are happy to consider but might take time to be released.
+
+## CI/CD pipelines
+
+The **continuous development** (CI/CD) tools of GitLab are used:
+
+- The [test suite](./tests.md) is run remotely when pushing new commits to the
+ 'develop' or to a release branch.
+- It is also triggered when pushing to a feature branch that is the object of
+ an open merge request that is not tagged to be a draft and that targets the
+ develop or a release branch.
+- It may be triggered manually for any merge request commit, whether draft or
+ not, via the online gitlab interface.
diff --git a/docs/devs-guide/docs-build.md b/docs/devs-guide/docs-build.md
new file mode 100644
index 0000000000000000000000000000000000000000..87d0eac0221601de62a57ab1c77f603a6d812c92
--- /dev/null
+++ b/docs/devs-guide/docs-build.md
@@ -0,0 +1,70 @@
+# Building the documentation
+
+The documentation rendered on our website is built automatically every time
+a new release of the package occurs. You may however want to build and render
+the website locally, notably to check that docstrings you wrote are properly
+parsed and rendered.
+
+## Generating markdown files
+
+The markdown documentation of declearn, based on which our website is rendered,
+is split between a static part (including the file you are currently reading),
+and a dynamically-generated one: namely, the full API reference, as well as
+part of the home index file.
+
+The `gen_docs.py` script may be used to generate the dynamically-created
+markdown files:
+
+- `docs/index.md` is created based on a hard-coded template and contents
+ parsed from the "README.md" file (to avoid discrepancies).
+- `docs/api-reference/` files and subfolders are generated procedurally
+ based on the exploration of the source code, using the `griffe` third-
+ party tool that `mkdocstrings` also makes use of.
+
+## Generating the website
+
+The website is automatically generated using [mkdocs](https://www.mkdocs.org/)
+and [mkdocstrings](https://mkdocstrings.github.io/) - the latter being used to
+render the minimal API reference markdown files into actual content parsed from
+the code's docstrings and type annotations.
+
+To build the website locally, you may use the following instructions:
+
+```bash
+# Clone the gitlab repository (optionally targetting a given branch).
+git clone https://gitlab.inria.fr/magnet/declearn/declearn2.git declearn
+cd declearn
+
+# Install the required dependencies (preferably in a dedicated venv).
+# You may find the up-to-date list of dependencies in the `pyproject.toml`
+# file, under [project.optional-dependencies], as the "docs" table.
+pip install \
+ mkdocstrings[python] \
+ mkdocs-autorefs \
+ mkdocs-literate-nav \
+ mkdocs-material
+
+# Auto-generate the API reference and home index markdown files.
+python scripts/gen_docs.py
+
+# Build the docs and serve them on your localhost.
+mkdocs build
+mkdocs serve # by default, serve on localhost:8000
+```
+
+In practice, the actual documentation website is built using
+[mike](https://github.com/jimporter/mike), so as to preserve access to the
+documentation of past releases. This is however out of scope for building
+and testing the documentation at a given code point locally.
+
+## Contributing to the documentation
+
+You may contribute changes to the out-of-code documentation by modifying
+static markdown files and opening a merge request, just as you would for
+source code modifications. Note that modifications to the home page should
+be contributed to the `gen_docs.py` script and/or `README.md` file; other
+markdown files' modifications may be proposed directly.
+
+Contributions should be pushed to the main repository (i.e. the one that
+holds the source code) rather than to the website one, as the latter is
+periodically updated by pulling from the former.
diff --git a/docs/devs-guide/docs-style.md b/docs/devs-guide/docs-style.md
new file mode 100644
index 0000000000000000000000000000000000000000..eaf3d386fc27338e5782e9329695c8aee8adadd4
--- /dev/null
+++ b/docs/devs-guide/docs-style.md
@@ -0,0 +1,313 @@
+# Docstrings style guide
+
+The declearn docstrings are written, and more importantly parsed, based on the
+[numpy](https://numpydoc.readthedocs.io/en/latest/format.html) style. Here are
+some details and advice as to the formatting, both for coherence within the
+codebase and for proper rendering (see the dedicated section above on
+[building the docs](#building-the-documentation)).
+
+## Public functions and methods
+
+For each public function or method, you should provide with:
+
+### Description
+
+As per [PEP 257](https://peps.python.org/pep-0257/), a function or method's
+docstring should start with a one-line short description, written in imperative
+style, and optionally followed by a multi-line description. The latter may be
+as long and detailed as required, and possibly comprise cross-references or
+be structured using [lists](#lists-rendering).
+
+### "Parameters" section
+
+If the function or method expects one or more (non-self or cls) argument,
+they should be documented as part of a "Parameters" section.
+
+Example:
+```
+Parameters
+----------
+toto:
+ Description of the "toto" parameter.
+babar:
+ Description of the "babar" parameter.
+```
+
+It is not mandatory to specify the types of the parameters or their
+default values, as they are already specified as part of the signature,
+and parsed from it by mkdocs. You may however do it, notably in long
+docstrings, or for clarity purposes on complex signatures.
+
+### "Returns" or "Yields" section
+
+If the function or method returns one or more values, they should be documented
+as part of a "Returns" section. For generators, use an equivalent "Yields"
+section.
+
+Example:
+```
+Returns
+-------
+result:
+ Description of the "result" output.
+```
+
+Notes:
+
+- Naming outputs, even when a single value is returned, is useful for
+ clarity purposes. Please choose names that are short yet unequivocal.
+- It is possible to specify the types of the returned values, but this
+ will override the type-hints in the mkdocs-rendered docs and should
+ therefore be used sparingly.
+
+### "Raises" section
+
+If some exceptions are raised or expected by the function, they should be
+documented as part of a "Raises" section.
+
+Example:
+```
+Raises
+------
+KeyError
+ Description of the reasons why a KeyError would be raised.
+TypeError
+ Description of the reasons why a TypeError would be raised.
+```
+
+Use this section sparingly: document errors that are raised as part
+of this function or method, and those raised by its backend code if
+any (notably private functions or methods it calls). However, unless
+you actually expect some to be raised, do not list all exceptions
+that may be raised by other components (e.g. FileNotFoundError in
+case an input string does not point to an actual file) if these are
+not the responsibility of declearn and/or should be obvious to the
+end-users.
+
+### "Warns" section
+
+If some warnigns are or may be emitted by the function, they should be
+documented as part of a "Warns" section.
+
+- Use the same format as for "Raises" sections.
+- You do not need to explicitly document deprecation warnings if the
+ rest of the docstring alread highlights deprecation information.
+- Similarly, some functions may automatically raise a warning, e.g.
+ because they trigger unsafe code: these warnings do not need to be
+ documented - but the unsafety does, as part of the main description.
+
+## Private functions and methods
+
+In general, private functions and methods should be documented following the
+same rules as the public ones. You may however sparingly relax the rules
+above, e.g. for simple, self-explanatory functions that may be described with
+a single line, or if a private method shares the signature of a public one,
+at which rate you may simply refer developers to the latter. As per the
+[Zen of Python](https://peps.python.org/pep-0020/), code readability and
+clarity should prevail.
+
+## Classes
+
+The main docstring of a public class should detail the following:
+
+### Description
+
+The formatting rules of the description are the same as for functions. A
+class's description may be very short, notably for children of base APIs
+that are already expensive as to how things work in general.
+
+For API-defining (abstract) base classes, an overview of the API should
+be provided, optionally structured into subsections.
+
+## "Attributes" section
+
+An "Attributes" section should detail the public attributes of class instances.
+
+Example:
+```
+Attributes
+----------
+toto:
+ Description of the toto attribute of instances.
+babar:
+ Description of the babar attribute of instances.
+```
+
+
+Historically, these have not been used in declearn half as much as they
+should. Efforts towards adding them are undergoing, and new code should
+not neglect this section.
+
+## Additional sections
+
+Optionally, one or more sections may be used to detail methods.
+
+Detailing methods is not mandatory and should preferably be done sparingly.
+Typically, you may highlight and/or group methods that are going to be of
+specific interest to end-users and/or developers, but do not need to list
+each and every method - as this can typically be done automatically, both
+by python's `help` function and mkdocs when rendering the docs.
+
+
+For such sections, you need to add listing ticks for mkdocs to parse the
+contents properly:
+```
+Key methods
+-----------
+- method_a:
+ Short description of the method `method_a`.
+- method_b:
+ Short description of the method `method_b`.
+```
+
+A typical pattern used for API-defining abstract bases classes in declearn
+is to use the following three sections:
+
+- "Abstract": overview of the abstract class attributes and methods.
+- "Overridable": overview of the key methods that children classes are
+ expected to overload or override to adjust their behavior.
+- "Inheritance": information related to types-registration mechanisms
+ and over inheritance-related details.
+
+## Constants and class attributes
+
+For constant variables and class attributes, you may optionally write up a
+docstring that will be rendered by mkdocs (but have no effect in python),
+by writing up a three-double-quotes string on the line(s) that follow(s)
+the declaration of the variable.
+
+Example:
+```
+MY_PUBLIC_CONSTANT = 42.0
+"""Description of what this constant is and where/how it is used."""
+```
+
+This feature should be used sparingly: currently, constants that have such
+a docstring attached will be exposed as part of the mkdocs-rendered docs,
+while those that do not will not be rendered. Hence it should be used for
+things that need to be exposed to the docs-consulting users; typically,
+abstract (or base-class-level) class attributes, or type-hinting aliases
+that are commonly used in some public function or method signatures.
+
+## Public modules' main docstring
+
+The main docstring of public modules (typically, that of `__init__.py` files)
+are a key component of the rendered documentation. They should detail, in an
+orderly manner, all the public functions, classes and submodules they expose,
+so as to provide with a clear overview of the module and its subcomponents.
+It should also contain cross-references to these objects, so that the mkdocs-
+rendered documentation turns their names into clickable links to the detailed
+documentation of these objects.
+
+Example:
+```
+"""Short description of the module.
+
+Longer description of the module.
+
+Category A
+----------
+- [Toto][declearn.module.Toto]:
+ Short description of exported class Toto.
+- [func][declearn.module.func]:
+ Short description of exported function func.
+
+Submodules
+----------
+- [foo][declearn.module.foo]:
+ Short description of public submodule foo.
+- [bar][declearn.module.bar]:
+ Short description of public submodule bar.
+"""
+```
+
+Note that the actual formatting of the description if open: sections may be
+used, as in the example above, to group contents (and add them to the rendered
+website page's table of contents), but you may also simply use un-named lists,
+or include the names of (some of) the objects and/or submodules as part of a
+sentence contextualizing what they are about.
+
+The key principles are:
+- Clarity and readability should prevail over all.
+- Readability should be two-fold: for python `help` users and for the rendered
+ website docs.
+- Provide links to everything that is exported at this level, so as to act as
+ an exhaustive summary of the module.
+- Optionally provide links to things that come from submodules, if it will
+ help end-users to make their way through the package.
+
+## General information and advice
+
+### Lists rendering
+
+To have a markdown-like listing properly rendered by mkdocs, you need to add
+a blank line above it:
+
+```
+My listing:
+
+- item a
+- item b
+```
+
+will be properly rendered, whereas
+
+```
+My listing:
+- item a
+- item b
+```
+will be unproperly rendered as `My listing: - item a - item b`
+
+For nested lists, make sure to indent by 4 spaces (not 2) so that items are
+rendered on the desired level.
+
+### Example code blocks
+
+Blocks of example code should be put between triple-backquote delimiters to
+be properly rendered (and avoid messing the rest of the docstrings' parsing).
+
+Example:
+``````
+Usage
+-----
+```
+>>> my_command(...)
+... expected output
+```
+
+Rest of the docstring.
+``````
+
+### Pseudo-code blocks
+
+You may render some parts of the documentation as unformatted content delimited
+by a grey-background block, by adding an empty line above _and_ identing that
+content. This may typically be used to render the pseudo-code description of
+algorithms:
+```
+"""Short description of the function.
+
+This function implements the following algorithm:
+
+ algorithm...
+ algorithm...
+ etc.
+
+Rest of the description, arguments, etc.
+"""
+```
+
+### LaTeX rendering
+
+You may include some LaTeX formulas as part of docstrings, that will be
+rendered using [mathjax](https://www.mathjax.org/) in the website docs,
+as long as they are delimited with `$$` signs, and you double all backquote
+to avoid bad python parsing (e.g. `$$ f(x) = \\sqrt{x} $$`).
+
+These should be used sparingly, to avoid making the raw docstrings unreadable
+by developers and python `help()` users. Typically, a good practice can be to
+explain the formula and/or provide with some pseudo-code, and then add the
+LaTeX formulas as a dedicated section or block (e.g. as an annex), so that
+they can be skipped when reading the raw docstring without missing other
+information than the formulas themselves.
diff --git a/docs/devs-guide/index.md b/docs/devs-guide/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..d730e01a772ed1f1fcbf5cdb7e713f7e7b7367d3
--- /dev/null
+++ b/docs/devs-guide/index.md
@@ -0,0 +1,16 @@
+# Developer Guide
+
+This guide is intended to provide developers with instructions as to how to
+contribute to declearn, what the coding and documentation style rules are,
+and which tools may or should be used to lint and test your code.
+
+This guide is structured this way:
+
+- [Contributions guide](./contribute.md):<br/>
+ Guide on our use of git and gitlab, and overview of our coding rules.
+- [Unit tests and code analysis](./tests.md):<br/>
+ Information on how to run the tests and lint your code.
+- [Building the documentation](./docs-build.md):<br/>
+ Guide on how to build and render the package's documentation.
+- [Docstrings style guide](./docs-style.md):<br/>
+ Guide on how to format docstrings and have them properly rendered.
diff --git a/docs/devs-guide/tests.md b/docs/devs-guide/tests.md
new file mode 100644
index 0000000000000000000000000000000000000000..f514d26943f51b19f453a2cff61563430eac0c7b
--- /dev/null
+++ b/docs/devs-guide/tests.md
@@ -0,0 +1,182 @@
+# Unit tests and code analysis
+
+Unit tests, as well as more-involved functional ones, are implemented under
+the `test/` folder of the declearn gitlab repository. Functional tests are
+isolated in the `test/functional/` subfolder to enable one to easily exclude
+them in order to run unit tests only.
+
+Tests are implemented using the [PyTest](https://docs.pytest.org) framework,
+as well as some third-party plug-ins that are automatically installed with
+the package when using `pip install declearn[tests]`.
+
+Additionally, code analysis tools are configured through the `pyproject.toml`
+file, and used to control code quality upon merging to the main branch. These
+tools are [black](https://github.com/psf/black) for code formatting,
+[pylint](https://pylint.pycqa.org/) for overall static code analysis and
+[mypy](https://mypy.readthedocs.io/) for static type-cheking.
+
+
+## Running the unit tests suite
+
+### Running the test suite using tox
+
+The third-party [tox](https://tox.wiki/en/latest/) tool may be used to run
+the entire test suite within a dedicated virtual environment. Simply run `tox`
+from the commandline with the root repo folder as working directory. You may
+optionally specify the python version(s) with which you want to run tests.
+
+```bash
+tox # run with default python 3.8
+tox -e py310 # override to use python 3.10
+```
+
+Note that additional parameters for `pytest` may be passed as well, by adding
+`--` followed by any set of options you want at the end of the `tox` command.
+For example, to use the declearn-specific `--fulltest` option (see the section
+below), run:
+
+```bash
+tox [tox options] -- --fulltest
+```
+
+The tests pipeline specified under the `tox.ini` file runs the following:
+
+- install declearn in an isolated environment
+- run unit tests
+- run functional tests
+- run pylint on declearn, then on the tests' code
+- run mypy on declearn
+- run black on declearn, in check mode
+
+### Running unit tests using pytest
+
+To run all the tests, simply use:
+
+```bash
+pytest test
+```
+
+To run the tests under a given module (here, "model"):
+
+```bash
+pytest test/model
+```
+
+To run the tests under a given file (here, "test_regression.py"):
+
+```bash
+pytest test/functional/test_regression.py
+```
+
+Note that by default, some test scenarios that are considered somewhat
+superfluous~redundant will be skipped in order to save time. To avoid
+skipping these, and therefore run a more complete test suite, add the
+`--fulltest` option to pytest:
+
+```bash
+pytest --fulltest test # or any more-specific target you want
+```
+
+For more details on how to run targetted tests, please refer to the
+[pytest](https://docs.pytest.org/) documentation.
+
+You may also arguments to compute and export coverage statistics, using the
+[pytest-cov](https://pytest-cov.readthedocs.io/en/latest/index.html) plug-in:
+
+```bash
+# Run all tests and export coverage information in HTML format.
+pytest --cov=declearn --cov-report=html tests/
+```
+
+## Running black to format the code
+
+The [black](https://github.com/psf/black) code formatter is used to enforce
+uniformity of the source code's formatting style. It is configured to have
+a maximum line length of 79 (as per [PEP 8](https://peps.python.org/pep-0008/))
+and ignore auto-generated protobuf files, but will otherwise modify files
+in-place when executing the following commands from the repository's root
+folder:
+
+```bash
+black declearn # reformat the package
+black test # reformat the tests
+```
+
+Note that it may also be called on individual files or folders.
+One may "blindly" run black, however it is actually advised to have a look
+at the reformatting operated, and act on any readability loss due to it. A
+couple of advice:
+
+1. Use `#fmt: off` / `#fmt: on` comments sparingly, but use them.
+<br/>It is totally okay to protect some (limited) code blocks from
+reformatting if you already spent some time and effort in achieving a
+readable code that black would disrupt. Please consider refactoring as
+an alternative (e.g. limiting the nest-depth of a statement).
+
+2. Pre-format functions and methods' signature to ensure style homogeneity.
+<br/>When a signature is short enough, black may attempt to flatten it as a
+one-liner, whereas the norm in declearn is to have one line per argument,
+all of which end with a trailing comma (for diff minimization purposes). It
+may sometimes be necessary to manually write the code in the latter style
+for black not to reformat it.
+
+Finally, note that the test suite run with tox comprises code-checking by
+black, and will fail if some code is deemed to require alteration by that
+tool. You may run this check manually:
+
+```bash
+black --check declearn # or any specific file or folder
+```
+
+## Running pylint to check the code
+
+The [pylint](https://pylint.pycqa.org/) linter is expected to be used for
+static code analysis. As a consequence, `# pylint: disable=[some-warning]`
+comments can be found (and added) to the source code, preferably with some
+indication as to the rationale for silencing the warning (or error).
+
+A minimal amount of non-standard hyper-parameters are configured via the
+`pyproject.toml` file and will automatically be used by pylint when run
+from within the repository's folder.
+
+Most code editors enable integrating the linter to analyze the code as it is
+being edited. To lint the entire package (or some specific files or folders)
+one may simply run `pylint`:
+
+```bash
+pylint declearn # analyze the package
+pylint test # analyze the tests
+```
+
+Note that the test suite run with tox comprises the previous two commands,
+which both result in a score associated with the analyzed code. If the score
+does not equal 10/10, the test suite will fail - notably preventing acceptance
+of merge requests.
+
+## Running mypy to type-check the code
+
+The [mypy](https://mypy.readthedocs.io/) linter is expected to be used for
+static type-checking code analysis. As a consequence, `# type: ignore` comments
+can be found (and added) to the source code, as sparingly as possible (mostly,
+to silence warnings about untyped third-party dependencies, false-positives,
+or locally on closure functions that are obvious enough to read from context).
+
+Code should be type-hinted as much and as precisely as possible - so that mypy
+actually provides help in identifying (potential) errors and mistakes, with
+code clarity as final purpose, rather than being another linter to silence off.
+
+A minimal amount of parameters are configured via the `pyproject.toml` file,
+and some of the strictest rules are disabled as per their default value (e.g.
+Any expressions are authorized - but should be used sparingly).
+
+Most code editors enable integrating the linter to analyze the code as it is
+being edited. To lint the entire package (or some specific files or folders)
+one may simply run `mypy`:
+
+```bash
+mypy declearn
+```
+
+Note that the test suite run with tox comprises the previous command. If mypy
+identifies errors, the test suite will fail - notably preventing acceptance
+of merge requests.
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..94f3aeffc2c6c34564d1555997e2597e1b263f08
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,85 @@
+# Declearn: a modular and extensible framework for Federated Learning
+
+## Introduction
+
+[declearn](https://magnet.gitlabpages.inria.fr/declearn/docs/2.0/)
+is a python package providing with a framework to perform federated
+learning, i.e. to train machine learning models by distributing computations
+across a set of data owners that, consequently, only have to share aggregated
+information (rather than individual data samples) with an orchestrating server
+(and, by extension, with each other).
+
+The aim of `declearn` is to provide both real-world end-users and algorithm
+researchers with a modular and extensible framework that:
+
+- builds on **abstractions** general enough to write backbone algorithmic code
+ agnostic to the actual computation framework, statistical model details
+ or network communications setup
+- designs **modular and combinable** objects, so that algorithmic features, and
+ more generally any specific implementation of a component (the model, network
+ protocol, client or server optimizer...) may easily be plugged into the main
+ federated learning process - enabling users to experiment with configurations
+ that intersect unitary features
+- provides with functioning tools that may be used **out-of-the-box** to set up
+ federated learning tasks using some popular computation frameworks (scikit-
+ learn, tensorflow, pytorch...) and federated learning algorithms (FedAvg,
+ Scaffold, FedYogi...)
+- provides with tools that enable **extending** the support of existing tools
+ and APIs to custom functions and classes without having to hack into the
+ source code, merely adding new features (tensor libraries, model classes,
+ optimization plug-ins, orchestration algorithms, communication protocols...)
+ to the party
+
+At the moment, `declearn` has been focused on so-called "centralized" federated
+learning that implies a central server orchestrating computations, but it might
+become more oriented towards decentralized processes in the future, that remove
+the use of a central agent.
+
+## Explore the documentation
+
+The documentation is structured this way:
+
+- [Installation guide](./setup.md):<br/>
+ Learn how to set up for and install declearn.
+- [Quickstart example](./quickstart.md):<br/>
+ See in a glance what end-user declearn code looks like.
+- [User guide](./user-guide/index.md):<br/>
+ Learn about declearn's take on Federated Learning, its current capabilities,
+ how to implement your own use case, and the API's structure and key points.
+- [API Reference](./api-reference/index.md):<br/>
+ Full API documentation, auto-generated from the source code.
+- [Developer guide](./devs-guide/index.md):<br/>
+ Information on how to contribute, codings rules and how to run tests.
+
+## Copyright
+
+Declearn is an open-source software developed by people from the
+[Magnet](https://team.inria.fr/magnet/) team at [Inria](https://www.inria.fr/).
+
+### Authors
+
+Current core developers are listed under the `pyproject.toml` file. A more
+detailed acknowledgement and history of authors and contributors to declearn
+can be found in the `AUTHORS` file.
+
+### License
+
+Declearn distributed under the Apache-2.0 license. All code files should
+therefore contain the following mention, which also applies to the present
+README file:
+```
+Copyright 2023 Inria (Institut National de la Recherche en Informatique
+et Automatique)
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+```
diff --git a/docs/quickstart.md b/docs/quickstart.md
new file mode 100644
index 0000000000000000000000000000000000000000..72b041025376b0a29cc152db0209fee50ca9e2c2
--- /dev/null
+++ b/docs/quickstart.md
@@ -0,0 +1,103 @@
+# Quickstart
+
+This section provides with demonstration code on how to run a simple federated
+learning task using declearn, that requires minimal adjustments to be run for
+real (mainly, to provide with a valid network configuration and actual data).
+
+You may find even more concrete examples on our gitlab repository
+[here](https://gitlab.inria.fr/magnet/declearn/declearn2/examples).
+The Heart UCI example may notably be run as-is, either locally or on a
+real-life network with minimal command-line parametrization.
+
+## Setting
+
+Here is a quickstart example on how to set up a federated learning process
+to learn a LASSO logistic regression model (using a scikit-learn backend)
+using pre-processed data, formatted as csv files with a "label" column,
+where each client has two files: one for training, the other for validation.
+
+Here, the code uses:
+
+- standard FedAvg strategy (SGD for local steps, averaging of updates weighted
+ by clients' training dataset size, no modifications of server-side updates)
+- 10 rounds of training, with 5 local epochs performed at each round and
+ 128-samples batch size
+- at least 1 and at most 3 clients, awaited for 180 seconds by the server
+- network communications using gRPC, on host "example.com" and port 8888
+
+Note that this example code may easily be adjusted to suit use cases, using
+other types of models, alternative federated learning algorithms and/or
+modifying the communication, training and validation hyper-parameters.
+Please refer to the [Hands-on usage](./user-guide/usage.md) section for a more
+detailed and general description of how to set up a federated learning
+task and process with declearn.
+
+## Server-side script
+
+```python
+import declearn
+
+model = declearn.model.sklearn.SklearnSGDModel.from_parameters(
+ kind="classifier", loss="log_loss", penalty="l1"
+)
+netwk = declearn.communication.NetworkServerConfig(
+ protocol="grpc", host="example.com", port=8888,
+ certificate="path/to/certificate.pem",
+ private_key="path/to/private_key.pem"
+)
+optim = declearn.main.config.FLOptimConfig.from_params(
+ aggregator="averaging",
+ client_opt=0.001,
+)
+server = declearn.main.FederatedServer(
+ model, netwk, optim, checkpoint="outputs"
+)
+config = declearn.main.config.FLRunConfig.from_params(
+ rounds=10,
+ register={"min_clients": 1, "max_clients": 3, "timeout": 180},
+ training={"n_epoch": 5, "batch_size": 128, "drop_remainder": False},
+)
+server.run(config)
+```
+
+## Client-side script
+
+```python
+import declearn
+
+netwk = declearn.communication.NetworkClientConfig(
+ protocol="grpc",
+ server_uri="example.com:8888",
+ name="client_name",
+ certificate="path/to/root_ca.pem"
+)
+train = declearn.dataset.InMemoryDataset(
+ "path/to/train.csv", target="label",
+ expose_classes=True # enable sharing of unique target values
+)
+valid = declearn.dataset.InMemoryDataset("path/to/valid.csv", target="label")
+client = declearn.main.FederatedClient(
+ netwk, train, valid, checkpoint="outputs"
+)
+client.run()
+```
+
+## Simulating this experiment locally
+
+To simulate the previous experiment on a single computer, you may set up
+network communications to go through the localhost, and resort to one of
+two possibilities:
+
+1. Run the server and client-wise scripts parallelly, e.g. in distinct
+ terminals.
+2. Use declearn-provided tools to run the server and clients' routines
+ concurrently using multiprocessing.
+
+While technically similar (both solutions resolve on isolating the agents'
+routines in separate python processes that communicate over the localhost),
+the second solution offers more practicality in terms of offering a single
+entrypoint for your experiment, and optionally automatically stopping any
+running agent in case one of the other has failed.
+To find out more about this solution, please have a look at the Heart UCI
+example [implemented here](https://gitlab.inria.fr/magnet/declearn/declearn2\
+-/tree/develop/examples/heart-uci).
diff --git a/docs/setup.md b/docs/setup.md
new file mode 100644
index 0000000000000000000000000000000000000000..547f44b1126135e50cdf39724d0c2d178a623cdf
--- /dev/null
+++ b/docs/setup.md
@@ -0,0 +1,109 @@
+# Installation guide
+
+## Requirements
+
+- python >= 3.8
+- pip
+
+Third-party requirements are specified (and automatically installed) as part
+of the installation process, and may be consulted from the `pyproject.toml`
+file.
+
+## Optional requirements
+
+Some third-party requirements are optional, and may not be installed. These
+are also specified as part of the `pyproject.toml` file, and may be divided
+into two categories:<br/>
+(a) dependencies of optional, applied declearn components (such as the PyTorch
+and Tensorflow tensor libraries, or the gRPC and websockets network
+communication backends) that are not imported with declearn by default<br/>
+(b) dependencies for developers, e.g. to run tests on the package (mainly
+pytest and some of its plug-ins), or building its documentation (with mkdocs)
+
+The second category is more developer-oriented, while the first may or may not
+be relevant depending on the use case to which you wish to apply `declearn`.
+
+In the `pyproject.toml` file, the `[project.optional-dependencies]` table
+`all` lists all dependencies from the first category, with additional tables
+redundantly listing them thematically, enabling end-users to cherry-pick the
+optional components they want to install. For developers, the "tests" and
+"docs" tables specify tooling dependencies.
+
+## Using a virtual environment (optional)
+
+It is generally advised to use a virtual environment, to avoid any dependency
+conflict between declearn and packages you might use in separate projects. To
+do so, you may for example use python's built-in
+[venv](https://docs.python.org/3/library/venv.html), or the third-party tool
+[conda](https://docs.conda.io/en/latest/).
+
+Venv instructions (example):
+
+```bash
+python -m venv ~/.venvs/declearn
+source ~/.venvs/declearn/bin/activate
+```
+
+Conda instructions (example):
+
+```bash
+conda create -n declearn python=3.8 pip
+conda activate declearn
+```
+
+_Note: at the moment, conda installation is not recommended, because the
+package's installation is made slightly harder due to some dependencies being
+installable via conda while other are only available via pip/pypi, which caninstall
+lead to dependency-tracking trouble._
+
+## Installation
+
+### Install from PyPI
+
+Stable releases of the package are uploaded to
+[PyPI](https://pypi.org/project/declearn/), enabling one to install with:
+
+```bash
+pip install declearn # optionally with version constraints and/or extras
+```
+
+### Install from source
+
+Alternatively, to install from source, one may clone the git repository (or
+download the source code from a release) and run `pip install .` from its
+root folder.
+
+```bash
+git clone git@gitlab.inria.fr:magnet/declearn/declearn.git
+cd declearn
+pip install . # or pip install -e .
+```
+
+### Install extra dependencies
+
+To also install optional requirements, add the name of the extras between
+brackets to the `pip install` command, _e.g._ running one of the following:
+
+```bash
+# Examples of cherry-picked installation instructions.
+pip install declearn[grpc] # install dependencies to use gRPC communications
+pip install declearn[torch] # install `declearn.model.torch` dependencies
+pip install declearn[tensorflow,torch] # install both tensorflow and torch
+
+# Instructions to install bundles of optional components.
+pip install declearn[all] # install all extra dependencies, save for testing
+pip install declearn[all,tests] # install all extra and testing dependencies
+```
+
+### Notes
+
+- If you are not using a virtual environment, select carefully the `pip` binary
+ being called (e.g. use `python -m pip`), and/or add a `--user` flag to the
+ pip command.
+- Developers may have better installing the package in editable mode, using
+ `pip install -e .` from the repository's root folder.
+- If you are installing the package within a conda environment, it may be
+ better to run `pip install --no-deps declearn` so as to only install the
+ package, and then to manually install the dependencies listed in the
+ `pyproject.toml` file, using `conda install` rather than `pip install`
+ whenever it is possible.
diff --git a/docs/user-guide/SUMMARY.md b/docs/user-guide/SUMMARY.md
new file mode 100644
index 0000000000000000000000000000000000000000..3685e94f57228eaf4277cdf501b37e11414894ca
--- /dev/null
+++ b/docs/user-guide/SUMMARY.md
@@ -0,0 +1,5 @@
+- [Introduction](./index.md)
+- [Overview of the Federated Learning process](./fl_process.md)
+- [Overview of the declearn API](./package.md)
+- [Hands-on usage](./usage.md)
+- [Local Differential Privacy capabilities](./local_dp.md)
diff --git a/docs/user-guide/fl_process.md b/docs/user-guide/fl_process.md
new file mode 100644
index 0000000000000000000000000000000000000000..85f9fd1d8bc8f70b9738e5371d07469735b7e2eb
--- /dev/null
+++ b/docs/user-guide/fl_process.md
@@ -0,0 +1,108 @@
+# Overview of the Federated Learning process
+
+This overview describes the way the `declearn.main.FederatedServer`
+and `declearn.main.FederatedClient` pair of classes implement the
+federated learning process. It is however possible to subclass
+these and/or implement alternative orchestrating classes to define
+alternative overall algorithmic processes - notably by overriding
+or extending methods that define the sub-components of the process
+exposed here.
+
+## Overall process orchestrated by the server
+
+- Initially:
+ - have the clients connect and register for training
+ - prepare model and optimizer objects on both sides
+- Iteratively:
+ - perform a training round
+ - perform an evaluation round
+ - decide whether to continue, based on the number of
+ rounds taken or on the evolution of the global loss
+- Finally:
+ - restore the model weights that yielded the lowest global loss
+ - notify clients that training is over, so they can disconnect
+ and run their final routine (e.g. save the "best" model)
+ - optionally checkpoint the "best" model
+ - close the network server and end the process
+
+## Detail of the process phases
+
+### Registration process
+
+- Server:
+ - open up registration (stop rejecting all received messages)
+ - handle and respond to client-emitted registration requests
+ - await criteria to have been met (exact or min/max number of clients
+ registered, optionally under a given timeout delay)
+ - close registration (reject future requests)
+- Client:
+ - gather metadata about the local training dataset
+ (_e.g._ dimensions and unique labels)
+ - connect to the server and send a request to join training,
+ including the former information
+ - await the server's response (retry after a timeout if the request
+ came in too soon, i.e. registration is not opened yet)
+- messaging : (JoinRequest <-> JoinReply)
+
+### Post-registration initialization
+
+- Server:
+ - validate and aggregate clients-transmitted metadata
+ - finalize the model's initialization using those metadata
+ - send the model, local optimizer and evaluation metrics specs to clients
+- Client:
+ - instantiate the model, optimizer and metrics based on server instructions
+- messaging: (InitRequest <-> GenericMessage)
+
+### (Optional) Local differential privacy setup
+
+- This step is optional; a flag in the InitRequest at the previous step
+ indicates to clients that it is to happen, as a secondary substep.
+- Server:
+ - send hyper-parameters to set up local differential privacy, including
+ dp-specific hyper-parameters and information on the planned training
+- Client:
+ - adjust the training process to use sample-wise gradient clipping and
+ add gaussian noise to gradients, implementing the DP-SGD algorithm
+ - set up a privacy accountant to monitor the use of the privacy budget
+- messaging: (PrivacyRequest <-> GenericMessage)
+
+### Training round
+
+- Server:
+ - select clients that are to participate
+ - send data-batching and effort constraints parameters
+ - send shared model trainable weights and (opt. client-specific) optimizer
+ auxiliary variables
+- Client:
+ - update model weights and optimizer auxiliary variables
+ - perform training steps based on effort constraints
+ - step: compute gradients over a batch; compute updates; apply them
+ - finally, send back the local model weights' updates and optimizer
+ auxiliary variables
+- messaging: (TrainRequest <-> TrainReply)
+- Server:
+ - unpack and aggregate clients' model weights updates into global updates
+ - unpack and process clients' optimizer auxiliary variables
+ - run global updates through the server's optimizer to modify and finally
+ apply them
+
+### Evaluation round
+
+- Server:
+ - select clients that are to participate
+ - send data-batching parameters and shared model trainable weights
+ - (_send effort constraints, unused for now_)
+- Client:
+ - update model weights
+ - perform evaluation steps based on effort constraints
+ - step: update evaluation metrics, including the model's loss, over a batch
+ - optionally checkpoint the model, local optimizer and evaluation metrics
+ - send results to the server: optionally prevent sharing detailed metrics;
+ always include the scalar validation loss value
+- messaging: (EvaluateRequest <-> EvaluateReply)
+- Server:
+ - aggregate local loss values into a global loss metric
+ - aggregate all other evaluation metrics and log their values
+ - optionally checkpoint the model, optimizer, aggregated evaluation
+ metrics and client-wise ones
diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..22a38c74ccd7bf7d4517d43c714d9f58186c6c81
--- /dev/null
+++ b/docs/user-guide/index.md
@@ -0,0 +1,16 @@
+# Usage of the Python API
+
+This user guide is intended to provide with a detailed overview of declearn,
+show-casing what the package does, how it is designed, and where to look for
+more details regarding both the declearn API and Federated Learning in general.
+
+This guide is structured this way:
+
+- [Overview of the Federated Learning process](./fl_process.md):<br/>
+ Description of the Federated Learning process implemented by declearn.
+- [Overview of the declearn API](./package.md):<br/>
+ Description of the declearn code's structure and of the main APIs.
+- [Hands-on usage](./usage.md):<br/>
+ Guide on how to set up your own federated learning task using declearn.
+- [Local Differential Privacy capabilities](./local_dp.md):<br/>
+ Description of the local-DP features of declearn.
diff --git a/docs/user-guide/local_dp.md b/docs/user-guide/local_dp.md
new file mode 100644
index 0000000000000000000000000000000000000000..16e0065f7d61e192dd3543328e9f2ab2741f423b
--- /dev/null
+++ b/docs/user-guide/local_dp.md
@@ -0,0 +1,98 @@
+# Local Differential Privacy
+
+## Basics
+
+`declearn` comes with the possibility to train models using local differential
+privacy, as described in the centralized case by Abadi et al, 2016,
+[Deep Learning with Differential Privacy](https://arxiv.org/abs/1607.00133).
+This means that training can provide per-client privacy guarantees with regard
+to the central server.
+
+In practice, this can be done by simply adding a privacy field to the config
+file, object or input dict to the `run` method of `FederatedServer`. Taking
+the Heart UCI example, one simply has one line to add to the server-side
+script (`examples/heart-uci/server.py`) in order to implement local DP,
+here using Renyi-DP with epsilon=5, delta=0.00001 and a sample-wise gradient
+clipping parameter that binds their euclidean norm below 3:
+
+```python
+# These are the last statements in the `run_server` function.
+run_cfg = FLRunConfig.from_params(
+ # The following lines come from the base example:
+ rounds=20,
+ register={"min_clients": nb_clients},
+ training={"batch_size": 30, "drop_remainder": False},
+ evaluate={"batch_size": 50, "drop_remainder": False},
+ early_stop={"tolerance": 0.0, "patience": 5, "relative": False},
+ # DP-specific instructions (in their absence, do not use local DP):
+ privacy={"accountant": "rdp", "budget": (5, 10e-5), "sclip_norm": 3},
+)
+server.run(run_cfg) # this is unaltered
+```
+
+This implementation can breach privacy garantees for some standard model
+architecture and training processes, see the _Warnings and limits_ section.
+
+## More details on the backend
+
+Implementing local DP requires to change four key elements, which are
+automatically handled in declearn based on the provided privacy configuration:
+
+- **Add a privacy accountant**. We use the `Opacus` library, to set up a
+privacy accountant. The accountant is used in two key ways :
+ - To calculate how much noise to add to the gradient at each trainig step
+ to provide an $`(\epsilon-\delta)`$-DP guarantee over the total number of
+ steps planned. This is where the heavily lifting is done, as estimating
+ the tighest bounds on the privacy loss is a non-trivial problem. We default
+ to the Renyi-DP accountant used in the original paper, but Opacus provides
+ an evolving list of options, since this is an active area of research. For
+ more details see the documentation of `declearn.main.utils.PrivacyConfig`.
+ - To keep track of the privacy budget spent as training progresses, in
+ particular in case of early stopping.
+
+- **Implement per-sample gradient clipping**. Clipping bounds the sensitivity
+of samples' contributions to model updates. It is performed using the
+`max_norm` parameter of `Model.compute_batch_gradients`.
+
+- **Implement noise-addition to applied gradients**. A gaussian noise with a
+tailored variance is drawn and added to the batch-averaged gradients based on
+which the local model is updated at each and every training step.
+
+- **Use Poisson sampling to draw batches**. This is done at the `Dataset` level
+using the `poisson` argument of `Dataset.generate_batches`.
+ - As stated in the Opacus documentation, "Minibatches should be formed by
+ uniform sampling, i.e. on each training step, each sample from the dataset
+ is included with a certain probability p. Note that this is different from
+ standard approach of dataset being shuffled and split into batches: each
+ sample has a non-zero probability of appearing multiple times in a given
+ epoch, or not appearing at all."
+ - For more details, see Zhu and Wang, 2019,
+ [Poisson Subsampled Renyi Differential Privacy](http://proceedings.mlr.press/v97/zhu19c/zhu19c.pdf)
+
+## Warnings and limits
+
+Under certain model and training specifications, two silent breaches of formal
+privacy guarantees can occur. Some can be handled automatically if working
+with `torch`, but need to be manually checked for in other frameworks.
+
+- **Neural net layers that breach DP**. Standard architectures can lead
+to information leaking between batch samples. Know examples include batch
+normalization layers, LSTM, and multi-headed attention modules. In `torch`,
+checking a module for DP-compliance can be done using Opacus, by running:
+
+ ```python
+ #given an NN.module to be tested
+ from opacus import PrivacyEngine
+ dp_compatible_module = PrivacyEngine.get_compatible_module(module)
+ ```
+
+- **Gradient accumulation**. This feature is not used in standard declearn
+models and training tools, but users that might try to write custom hacks
+to simulate large batches by setting a smaller batch size and executing the
+optimization step every N steps over the accumulated sum of output gradients
+should be aware that this is not compatible with Poisson sampling.
+
+Finally, note that at this stage the DP implementation in declearn is taken
+directly from the centralized training case, and as such does not account for
+nor make use of some specifities of the Federated Learning process, such as
+privacy amplification by iteration.
diff --git a/docs/user-guide/package.md b/docs/user-guide/package.md
new file mode 100644
index 0000000000000000000000000000000000000000..4ac2de145d0afb14d727cd67ea6a4b1172ad295c
--- /dev/null
+++ b/docs/user-guide/package.md
@@ -0,0 +1,127 @@
+# Overview of the declearn API
+
+## Package structure
+
+The package is organized into the following submodules:
+
+- `aggregator`:<br/>
+   Model updates aggregating API and implementations.
+- `communication`:<br/>
+   Client-Server network communications API and implementations.
+- `data_info`:<br/>
+   Tools to write and extend shareable metadata fields specifications.
+- `dataset`:<br/>
+   Data interfacing API and implementations.
+- `main`:<br/>
+   Main classes implementing a Federated Learning process.
+- `metrics`:<br/>
+   Iterative and federative evaluation metrics computation tools.
+- `model`:<br/>
+   Model interfacing API and implementations.
+- `optimizer`:<br/>
+   Framework-agnostic optimizer and algorithmic plug-ins API and tools.
+- `typing`:<br/>
+   Type hinting utils, defined and exposed for code readability purposes.
+- `utils`:<br/>
+   Shared utils used (extensively) across all of declearn.
+
+## Main abstractions
+
+This section lists the main abstractions implemented as part of
+`declearn`, exposing their main object and usage, some examples
+of ready-to-use implementations that are part of `declearn`, as
+well as references on how to extend the support of `declearn`
+backend (notably, (de)serialization and configuration utils) to
+new custom concrete implementations inheriting the abstraction.
+
+### `Model`
+- Import: `declearn.model.api.Model`
+- Object: Interface framework-specific machine learning models.
+- Usage: Compute gradients, apply updates, compute loss...
+- Examples:
+ - `declearn.model.sklearn.SklearnSGDModel`
+ - `declearn.model.tensorflow.TensorflowModel`
+ - `declearn.model.torch.TorchModel`
+- Extend: use `declearn.utils.register_type(group="Model")`
+
+### `Vector`
+- Import: `declearn.model.api.Vector`
+- Object: Interface framework-specific data structures.
+- Usage: Wrap and operate on model weights, gradients, updates...
+- Examples:
+ - `declearn.model.sklearn.NumpyVector`
+ - `declearn.model.tensorflow.TensorflowVector`
+ - `declearn.model.torch.TorchVector`
+- Extend: use `declearn.model.api.register_vector_type`
+
+### `OptiModule`
+- Import: `declearn.optimizer.modules.OptiModule`
+- Object: Define optimization algorithm bricks.
+- Usage: Plug into a `declearn.optimizer.Optimizer`.
+- Examples:
+ - `declearn.optimizer.modules.AdagradModule`
+ - `declearn.optimizer.modules.MomentumModule`
+ - `declearn.optimizer.modules.ScaffoldClientModule`
+ - `declearn.optimizer.modules.ScaffoldServerModule`
+- Extend:
+ - Simply inherit from `OptiModule` (registration is automated).
+ - To avoid it, use `class MyModule(OptiModule, register=False)`.
+
+### `Regularizer`
+- Import: `declearn.optimizer.modules.Regularizer`
+- Object: Define loss-regularization terms as gradients modifiers.
+- Usage: Plug into a `declearn.optimizer.Optimizer`.
+- Examples:
+ - `declearn.optimizer.regularizer.FedProxRegularizer`
+ - `declearn.optimizer.regularizer.LassoRegularizer`
+ - `declearn.optimizer.regularizer.RidgeRegularizer`
+- Extend:
+ - Simply inherit from `Regularizer` (registration is automated).
+ - To avoid it, use `class MyRegularizer(Regularizer, register=False)`.
+
+### `Metric`
+- Import: `declearn.metrics.Metric`
+- Object: Define evaluation metrics to compute iteratively and federatively.
+- Usage: Compute local and federated metrics based on local data streams.
+- Examples:
+ - `declearn.metric.BinaryRocAuc`
+ - `declearn.metric.MeanSquaredError`
+ - `declearn.metric.MuticlassAccuracyPrecisionRecall`
+- Extend:
+ - Simply inherit from `Metric` (registration is automated).
+ - To avoid it, use `class MyMetric(Metric, register=False)`
+
+### `NetworkClient`
+- Import: `declearn.communication.api.NetworkClient`
+- Object: Instantiate a network communication client endpoint.
+- Usage: Register for training, send and receive messages.
+- Examples:
+ - `declearn.communication.grpc.GrpcClient`
+ - `declearn.communication.websockets.WebsocketsClient`
+- Extend:
+ - Simply inherit from `NetworkClient` (registration is automated).
+ - To avoid it, use `class MyClient(NetworkClient, register=False)`.
+
+### `NetworkServer`
+- Import: `declearn.communication.api.NetworkServer`
+- Object: Instantiate a network communication server endpoint.
+- Usage: Receive clients' requests, send and receive messages.
+- Examples:
+ - `declearn.communication.grpc.GrpcServer`
+ - `declearn.communication.websockets.WebsocketsServer`
+- Extend:
+ - Simply inherit from `NetworkServer` (registration is automated).
+ - To avoid it, use `class MyServer(NetworkServer, register=False)`.
+
+### `Dataset`
+- Import: `declearn.dataset.Dataset`
+- Object: Interface data sources agnostic to their format.
+- Usage: Yield (inputs, labels, weights) data batches, expose metadata.
+- Examples:
+ - `declearn.dataset.InMemoryDataset`
+- Extend: use `declearn.utils.register_type(group="Dataset")`
+
+## Full API Reference
+
+The full API reference, which is generated automatically from the code's
+internal documentation, can be found [here](../api-reference/index.md).
diff --git a/docs/user-guide/usage.md b/docs/user-guide/usage.md
new file mode 100644
index 0000000000000000000000000000000000000000..c9bdb1fe7f72f235b6cbc0bedc51518d4d7674ad
--- /dev/null
+++ b/docs/user-guide/usage.md
@@ -0,0 +1,144 @@
+# Hands-on usage
+
+Here are details on how to set up server-side and client-side programs
+that will run together to perform a federated learning process. Generic
+remarks from the [Quickstart](#quickstart) section hold here as well, the
+former section being an overly simple exemplification of the present one.
+
+You can follow along on a concrete example that uses the UCI heart disease
+dataset, that is stored in the `examples/uci-heart` folder. You may refer
+to the `server.py` and `client.py` example scripts, that comprise comments
+indicating how the code relates to the steps described below. For further
+details on this example and on how to run it, please refer to its own
+`readme.md` file.
+
+## Server setup instructions
+
+**1. Define a Model**
+
+ - Set up a machine learning model in a given framework
+ (_e.g._ a `torch.nn.Module`).
+ - Select the appropriate `declearn.model.api.Model` subclass to wrap it up.
+ - Either instantiate the `Model` or provide a JSON-serialized configuration.
+
+**2. Define a FLOptimConfig**
+
+ - Select a `declearn.aggregator.Aggregator` (subclass) instance to define
+ how clients' updates are to be aggregated into global-model updates on
+ the server side.
+ - Parameterize a `declearn.optimizer.Optimizer` (possibly using a selected
+ pipeline of `declearn.optimizer.modules.OptiModule` plug-ins and/or a
+ pipeline of `declearn.optimizer.regularizers.Regularizer` ones) to be
+ used by clients to derive local step-wise updates from model gradients.
+ - Similarly, parameterize an `Optimizer` to be used by the server to
+ (optionally) refine the aggregated model updates before applying them.
+ - Wrap these three objects into a `declearn.main.config.FLOptimConfig`,
+ possibly using its `from_config` method to specify the former three
+ components via configuration dicts rather than actual instances.
+ - Alternatively, write up a TOML configuration file that specifies these
+ components (note that 'aggregator' and 'server_opt' have default values
+ and may therefore be left unspecified).
+
+**3. Define a communication server endpoint**
+
+ - Select a communication protocol (_e.g._ "grpc" or "websockets").
+ - Select the host address and port to use.
+ - Preferably provide paths to PEM files storing SSL-required information.
+ - Wrap this into a config dict or use `declearn.communication.build_server`
+ to instantiate a `declearn.communication.api.NetworkServer` to be used.
+
+**4. Instantiate and run a FederatedServer**
+
+ - Instantiate a `declearn.main.FederatedServer`:
+ - Provide the Model, FLOptimConfig and Server objects or configurations.
+ - Optionally provide a MetricSet object or its specs (i.e. a list of
+ Metric instances, identifier names of (name, config) tuples), that
+ defines metrics to be computed by clients on their validation data.
+ - Optionally provide the path to a folder where to write output files
+ (model checkpoints and global loss history).
+ - Instantiate a `declearn.main.config.FLRunConfig` to specify the process:
+ - Maximum number of training and evaluation rounds to run.
+ - Registration parameters: exact or min/max number of clients to have
+ and optional timeout delay spent waiting for said clients to join.
+ - Training parameters: data-batching parameters and effort constraints
+ (number of local epochs and/or steps to take, and optional timeout).
+ - Evaluation parameters: data-batching parameters and effort constraints
+ (optional maximum number of steps (<=1 epoch) and optional timeout).
+ - Early-stopping parameters (optionally): patience, tolerance, etc. as
+ to the global model loss's evolution throughout rounds.
+ - Local Differential-Privacy parameters (optionally): (epsilon, delta)
+ budget, type of accountant, clipping norm threshold, RNG parameters.
+ - Alternatively, write up a TOML configuration file that specifies all of
+ the former hyper-parameters.
+ - Call the server's `run` method, passing it the former config object,
+ the path to the TOML configuration file, or dictionaries of keyword
+ arguments to be parsed into a `FLRunConfig` instance.
+
+## Clients setup instructions
+
+**1. Interface training data**
+
+ - Select and parameterize a `declearn.dataset.Dataset` subclass that
+ will interface the local training dataset.
+ - Ensure its `get_data_specs` method exposes the metadata that is to
+ be shared with the server (and nothing else, to prevent data leak).
+
+**2. Interface validation data (optional)**
+
+ - Optionally set up a second Dataset interfacing a validation dataset,
+ used in evaluation rounds. Otherwise, those rounds will be run using
+ the training dataset - which can be slow and/or lead to overfitting.
+
+**3. Define a communication client endpoint**
+
+ - Select the communication protocol used (_e.g._ "grpc" or "websockets").
+ - Provide the server URI to connect to.
+ - Preferable provide the path to a PEM file storing SSL-required information
+ (matching those used on the Server side).
+ - Wrap this into a config dict or use `declearn.communication.build_client`
+ to instantiate a `declearn.communication.api.NetworkClient` to be used.
+
+**4. Run any necessary import statement**
+
+ - If optional or third-party dependencies are known to be required, import
+ them (_e.g._ `import declearn.model.torch`).
+ - Read more about this point [below](#dependency-sharing).
+
+**5. Instantiate a FederatedClient and run it**
+
+ - Instantiate a `declearn.main.FederatedClient`:
+ - Provide the NetworkClient and Dataset objects or configurations.
+ - Optionally specify `share_metrics=False` to prevent sharing evaluation
+ metrics (apart from the aggregated loss) with the server out of privacy
+ concerns.
+ - Optionally provide the path to a folder where to write output files
+ (model checkpoints and local loss history).
+ - Call the client's `run` method and let the magic happen.
+
+## Logging
+
+Note that this section and the quickstart example both left apart the option
+to configure logging associated with the federated client and server, and/or
+the network communication handlers they make use of. One may simply set up
+custom `logging.Logger` instances and pass them as arguments to the class
+constructors to replace the default, console-only, loggers.
+
+The `declearn.utils.get_logger` function may be used to facilitate the setup
+of such logger instances, defining their name, verbosity level, and whether
+messages should be logged to the console and/or to an output file.
+
+## Dependency sharing
+
+One important issue that is not currently handled by declearn itself is that
+of ensuring that clients have loaded all dependencies that may be required
+to unpack the Model and Optimizer instances transmitted at initialization.
+At the moment, it is therefore left to users to agree on the dependencies
+that need to be imported as part of the client-side launching script.
+
+For example, if the trained model is an artificial neural network that uses
+PyTorch as implementation backend, clients will need to add the
+`import declearn.model.torch` statement in their code (and, obviously, to
+have `torch` installed). Similarly, if a custom declearn `OptiModule` was
+written to modify the way updates are computed locally by clients, it will
+need to be shared with clients - either as a package to be imported (like
+torch previously), or as a bit of source code to add on top of the script.
diff --git a/mkdocs.yml b/mkdocs.yml
new file mode 100644
index 0000000000000000000000000000000000000000..cf7c1870788cea3cc6156035f43085b36f25047d
--- /dev/null
+++ b/mkdocs.yml
@@ -0,0 +1,46 @@
+site_name: Declearn
+site_url: https://magnet.gitlabpages.inria.fr/declearn/docs
+site_dir: public
+
+site_description: >-
+ Declearn: a modular and extensible framework for Federated Learning
+
+theme:
+ name: material
+ icon:
+ repo: fontawesome/brands/git-alt
+
+repo_url: https://gitlab.inria.fr/magnet/declearn/declearn2
+repo_name: magnet/declearn
+
+markdown_extensions:
+ - pymdownx.arithmatex: # mathjax
+ generic: true
+
+extra:
+ version:
+ provider: mike
+ canonical_version: latest
+
+extra_javascript:
+ # mathjax
+ - https://polyfill.io/v3/polyfill.min.js?features=es6
+ - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js
+
+plugins:
+- autorefs
+- search:
+ lang: en
+- literate-nav:
+ nav_file: SUMMARY.md
+- mkdocstrings:
+ default_handler: python
+ handlers:
+ python:
+ options:
+ disable_private: true
+ docstring_style: numpy
+ docstring_section_style: "table"
+ show_if_no_docstring: false
+ show_root_toc_entry: true
+ show_signature_annotations: false
diff --git a/pyproject.toml b/pyproject.toml
index be3bc4cb5ad2a1885477d8f1014274e2bd5a23fe..e06f8e02048fbd056fb9ffd129b2f0717169ed4f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -44,7 +44,8 @@ dependencies = [
]
[project.optional-dependencies]
-all = [ # all non-tests extra dependencies
+# all non-docs, non-tests extra dependencies
+all = [
"functorch",
"grpcio >= 1.45",
"opacus ~= 1.1",
@@ -53,6 +54,7 @@ all = [ # all non-tests extra dependencies
"torch ~= 1.10",
"websockets ~= 10.1",
]
+# thematically grouped dependencies (part of "all")
dp = [
"opacus ~= 1.1",
]
@@ -64,31 +66,30 @@ tensorflow = [
"tensorflow ~= 2.5",
]
torch = [
- "functorch", # note: functorch is included with torch>=1.13
+ "functorch", # note: functorch is internalized by torch 2.0
"torch ~= 1.10",
]
websockets = [
"websockets ~= 10.1",
]
+# docs-building dependencies
+docs = [
+ "mkdocstrings[python] >= 0.8",
+ "mkdocs-autorefs",
+ "mkdocs-literate-nav",
+ "mkdocs-material >= 9.1",
+]
+# test-specific dependencies
tests = [
- # test-specific dependencies
"black ~= 23.0",
"mypy ~= 1.0",
"pylint >= 2.14",
"pytest >= 6.1",
"pytest-asyncio",
- # other extra dependencies (copy of "all")
- "functorch",
- "grpcio >= 1.45",
- "opacus ~= 1.1",
- "protobuf >= 3.19",
- "tensorflow ~= 2.5",
- "torch ~= 1.10",
- "websockets ~= 10.1",
]
[project.urls]
-homepage = "https://gitlab.inria.fr/magnet/declearn/declearn2"
+homepage = "https://magnet.gitlabpages.inria/declearn/docs"
repository = "https://gitlab.inria.fr/magnet/declearn/declearn2.git"
[tool.black]
diff --git a/scripts/gen_docs.py b/scripts/gen_docs.py
new file mode 100644
index 0000000000000000000000000000000000000000..0d44806fe7563f1a21996cf4d4a2502fad09fc42
--- /dev/null
+++ b/scripts/gen_docs.py
@@ -0,0 +1,144 @@
+# coding: utf-8
+
+# Copyright 2023 Inria (Institut National de Recherche en Informatique
+# et Automatique)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Script to auto-generate the docs' markdown files."""
+
+import os
+import re
+import shutil
+from typing import Dict, Tuple
+
+import griffe
+
+
+ROOT_FOLDER = os.path.split(os.path.dirname(os.path.abspath(__file__)))[0]
+DOCS_INDEX = """{title}
+
+## Introduction
+
+{intro}
+
+## Explore the documentation
+
+The documentation is structured this way:
+
+- [Installation guide](./setup.md):<br/>
+ Learn how to set up for and install declearn.
+- [Quickstart example](./quickstart.md):<br/>
+ See in a glance what end-user declearn code looks like.
+- [User guide](./user-guide/index.md):<br/>
+ Learn about declearn's take on Federated Learning, its current capabilities,
+ how to implement your own use case, and the API's structure and key points.
+- [API Reference](./api-reference/index.md):<br/>
+ Full API documentation, auto-generated from the source code.
+- [Developer guide](./devs-guide/index.md):<br/>
+ Information on how to contribute, codings rules and how to run tests.
+
+## Copyright
+
+{rights}
+"""
+
+
+
+def generate_index():
+ """Fill-in the main index file based on the README one."""
+ # Parse contents from the README file and write up the index.md one.
+ title, readme = _parse_readme()
+ # FUTURE: parse the existing index.md and fill it rather then overwrite?
+ docidx = DOCS_INDEX.format(
+ title=title,
+ intro=readme["Introduction"],
+ rights=readme["Copyright"],
+ )
+ # Write up the index.md file.
+ path = os.path.join(ROOT_FOLDER, "docs", "index.md")
+ with open(path, "w", encoding="utf-8") as file:
+ file.write(docidx)
+
+
+def _parse_readme() -> Tuple[str, Dict[str, str]]:
+ """Parse contents from the declearn README file."""
+ path = os.path.join(ROOT_FOLDER, "README.md")
+ with open(path, "r", encoding="utf-8") as file:
+ text = file.read()
+ title, text = text.split("\n", 1)
+ content = re.split(r"\n(## \w+\n+)", text)
+ readme = dict(zip(content[1::2], content[2::2]))
+ readme = {k.strip("# \n"): v.strip("\n") for k, v in readme.items()}
+ return title, readme
+
+
+def generate_api_docs():
+ """Auto-generate the API Reference docs' markdown files."""
+ # Build the API reference docs folder.
+ docdir = os.path.join(ROOT_FOLDER, "docs")
+ os.makedirs(docdir, exist_ok=True)
+ docdir = os.path.join(docdir, "api-reference")
+ if os.path.isdir(docdir):
+ shutil.rmtree(docdir)
+ os.makedirs(docdir)
+ # Recursively generate the module-wise files.
+ module = griffe.load(
+ os.path.join(ROOT_FOLDER, "declearn"),
+ submodules=True,
+ try_relative_path=True,
+ )
+ parse_module(module, docdir, root=True)
+
+
+def parse_module(
+ module: griffe.dataclasses.Module,
+ docdir: str,
+ root: bool = False,
+) -> str:
+ """Recursively auto-generate markdown files for a module."""
+ # Case of file-based public module (`module.py`).
+ if not module.is_init_module:
+ path = os.path.join(docdir, f"{module.name}.md")
+ with open(path, "w", encoding="utf-8") as file:
+ file.write(f"::: {module.path}")
+ return f"{module.name}.md"
+ # Case of folder-based public module (`module/`)
+ # Create a dedicated folder.
+ if not root: # skip for the main folder
+ docdir = os.path.join(docdir, module.name)
+ os.makedirs(docdir)
+ # Recursively create folders and files for public submodules.
+ pub_mod = {}
+ for key, mod in module.modules.items():
+ if not key.startswith("_"):
+ pub_mod[key] = parse_module(mod, docdir)
+ # Create files for classes and functions exported from private submodules.
+ for key, obj in module.members.items():
+ if obj.is_module or obj.module.name in pub_mod or key.startswith("_"):
+ continue
+ if not (obj.docstring or obj.is_class or obj.is_function):
+ continue
+ path = os.path.join(docdir, f"{obj.name}.md")
+ with open(path, "w", encoding="utf-8") as file:
+ file.write(f"#`{obj.path}`\n::: {obj.path}")
+ # Write up an overview file based on the '__init__.py' docs.
+ path = os.path.join(docdir, "index.md")
+ with open(path, "w", encoding="utf-8") as file:
+ file.write(f"::: {module.path}")
+ return f"{module.name}/index.md"
+
+
+if __name__ == "__main__":
+ generate_index()
+ generate_api_docs()
diff --git a/tox.ini b/tox.ini
index 3dd74d8e95702a9be15c0d634002ff1f4ad1131c..3053573ed7608cf5aa412aa54fee679418f0965b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -6,7 +6,7 @@ minversion = 3.18.0
[testenv]
recreate = True
extras =
- tests
+ all,tests
allowlist_externals =
openssl
commands=