Revise network communication backend and message-parsing logic.

Summary

This MR introduces some important backend changes to the communication and messaging APIs of DecLearn, resulting in more robust code (that is also easier to test and maintain), more efficient message parsing (that delays possibly-costly de-serialization to a time posterior to validity verification, both in the message-handling backend and application code) and the free extensibility of application messages, enabling to easily define and use custom message structures in downstream applications.

This MR follows the new version policy introduced for the upcoming v2.4 release: end-user code should remain workable, however some API details have changed in a breaking manner (namely, network communication endpoints now return SerializedMessage instances rather than Message ones; and metadata sharing as part of the FL process is now optional, delayed and more parsimonious).

API changes

• Introduce a new declearn.communication.api.backend submodule:

  • Introduce a new ActionMessage minimal API under its actions submodule, that defines hard-coded, lightweight and easy-to-parse data structures designed to convey information and content across network communications agnostic to the content's nature.
  • Revise and expose the MessagesHandler util, that now builds on the ActionMessage API to model remote calls and answer them.
  • Move the declearn.communication.messaging.flags submodule to declearn.communication.api.backend.flags.

• Introduce a new declearn.messaging submodule:

  • Revise the Message API to make it extendable, with automated type-registration of subclasses by default.
  • Introduce SerializedMessage as a wrapper for received messages, that parses the exact Lessage subtype (enabling logic tests and message filtering) but delays actual content de-serialization and Message object recovery (enabling to prevent undue resources use for unwanted messages that end up being discarded).
  • Move most existing Message subclasses to the new submodule, for retro-compatibility purposes. In DecLearn 3.0 these will probably be re-dispatched to make it clear that concrete messages only make sense in the context of specific multi-agent processes.
  • Drop backend-oriented Message subclasses that are replaced with the new ActionMessage backbone structures.
  • Deprecate the declearn.communication.messaging submodule, that is temporarily maintained, re-exporting moved contents as well as deprecated message types (which are bound to be rejected if sent).

• Revise NetworkClient and NetworkServer:

  • Have message-receiving methods return SerializedMessage instances rather than finalized de-serialized Message ones.
  • Quit sending and expecting 'data_info' with registration requests.
  • Rename NetworkClient.check_message into recv_message (keep the former as an alias, with a DeprecationWarning).
  • Improve the use of (optional) timeouts when sending or expecting messages and overall exceptions handling:
    • NetworkClient.recv_message may either raise a TimeoutError (in case of timeout) or RuntimeError (in case of rejection).
    • NeworkServer.send_messages and broadcast_message quietly stop waiting for clients to collect messages after the (opt.) timeout delay has passed. Messages may still be collected.
    • NetworkServer.wait_for_messages no longer accepts a timeout.
    • NetworkServer.wait_for_messages_with_timeout implements the possibility to setup a timeout. It returns both received client replies and a list of clients that failed to answer.
    • All timeouts can now be specified as float values (which is mostly useful for testing purposes or simulated environments).
  • Add a heartbeat instantiation parameter, with a default value of 1 second, that is passed to the underlying MessagesHandler. In simulated contexts (including tests), setting a low heartbeat can cut runtime down significantly.

Notes

• Some declearn.communication.api.backend.actions structures were implemented merely for the sake of ensuring that requests received from older DecLearn clients would be detected as legacy and answered to accordingly (rejected with an instruction to update the version).
• Thanks to the introduction of heartbeat, unit and functional tests can now run more quickly when communications are involved.
• On the side, functional tests for the quickrun mode were fixed. They had been running without actually executing for some time.

added api-change backend tests labels

assigned to @paandrey

A note on unit tests:

• I implemented full-coverage unit tests on the new declearn.communication.api.backend submodule, and am rather happy with the ones tackling MessagesHandler.
• For existing tests, I merely updated them due to the API changes, but I did not trim parts that are in fact (partly) redundant with the new MessagesHandler unit tests, nor did I go into the effort of writing new "truly-unit" tests that would mock the underlying handler for the sake of testing that the code calls it as it should. It would be a tedious effort, for no gain in coverage and barely a gain in test runtime.
• I however put some effort into adding some fail-safes for potentially never-stopping processes. I may not have been exhaustive about it, but basically every test that I had to revise that proved to result in an infinite loop before was (hopefully) made safe by adding some timeout instructions.

added 2 commits

• dde0a48c - Add unit tests for 'declearn.communication.api.backend' code.
• 59b09c2e - Add 'heartbeat' parameter to 'NetworkServer' and speed up tests.

Compare with previous version

added 7 commits

• b2955133 - Adjust 'FederatedClient' and 'FederatedServer' to new messaging API.
• a407ec06 - Fix detection and rejection of legacy clients' messages.
• 19963de0 - Update unit tests for 'communication' module.
• 54a80493 - Enable specifying message-related timeouts as floats.
• a4be8084 - Turn version-based rejection into a (version-based) constant flag.
• 1a79cdc6 - Add unit tests for 'declearn.communication.api.backend' code.
• e8d04fcb - Add 'heartbeat' parameter to 'NetworkServer' and speed up tests.

Compare with previous version

changed the description

added 11 commits

• 7c65a283 - 1 commit from branch develop
• 7b623317 - Revise network communication backend and message-parsing logic.
• 42153810 - Revise 'Message' API, make it extendable and move it around.
• 125e2708 - Integrate new Message API with communication endpoints.
• 3b8703f2 - Adjust 'FederatedClient' and 'FederatedServer' to new messaging API.
• 1f554f14 - Fix detection and rejection of legacy clients' messages.
• bbe43fba - Update unit tests for 'communication' module.
• 7937bee7 - Enable specifying message-related timeouts as floats.
• 39543659 - Turn version-based rejection into a (version-based) constant flag.
• 07ed21db - Add unit tests for 'declearn.communication.api.backend' code.
• f770f370 - Add 'heartbeat' parameter to 'NetworkServer' and speed up tests.

Compare with previous version

enabled an automatic merge when the pipeline for f770f370 succeeds

mentioned in commit 18a459fe

merged