bug in integration test
2 bugs detected in integration tests:
-
tests launch notebooks that require another dataset than MNIST, but don't load this dataset on the node, thus notebook execution fails (see below) -
pytorch-celeba-dataset.py
requires dataset CELEBA -
pytorch-csv-data.py
requires a CSV dataset
-
-
job succeeds
Error log excerpt:
# 2023-02-08 09:00:13,326 fedbiomed INFO - Sampled nodes in round 0 []
# 2023-02-08 09:00:13,327 fedbiomed INFO - Nodes that successfully reply in round 0 []
# 2023-02-08 09:00:13,327 fedbiomed CRITICAL - Fed-BioMed stopped due to unknown error:
# An empty list of models was passed.
#
# --------------------
# Fed-BioMed researcher stopped due to unknown error:
# An empty list of models was passed.
# More details in the backtrace extract below
# --------------------
# Traceback (most recent call last):
# File "/builds/workspace/fedbiomed-integration-linux/fedbiomed/researcher/experiment.py", line 63, in payload
# ret = function(*args, **kwargs)
# File "/builds/workspace/fedbiomed-integration-linux/fedbiomed/researcher/experiment.py", line 1606, in run_once
# aggregated_params = self._aggregator.aggregate(model_params,
# File "/builds/workspace/fedbiomed-integration-linux/fedbiomed/researcher/aggregators/fedavg.py", line 40, in aggregate
# return federated_averaging(model_params_processed, weights_processed)
# File "/builds/workspace/fedbiomed-integration-linux/fedbiomed/researcher/aggregators/functional.py", line 33, in federated_averaging
# assert len(model_params) > 0, 'An empty list of models was passed.'
# AssertionError: An empty list of models was passed.
# --------------------
...
## Test is failed for this notebook
# == Failure with status: 3
ok 7 pytorch-local-training.py # in 111799 ms
...
## ... but CI job succeeds (return code from notebook execution is not tested ?)
# ======================================================================
# == quitting
# ======================================================================
# == Success
+ ansifilter report.tap -o report.filtered.tap
+ /bin/bash -c 'sed '\''s/^===/# ===/'\'' report.filtered.tap > report.filtered.patched.tap'
TAP Reports Processing: START
Looking for TAP results report in workspace using pattern: report.filtered.patched.tap
Saving reports...
Processing '/net/www/ci/fedbiomed/jobs/fedbiomed-integration-linux/builds/112/tap-master-files/report.filtered.patched.tap'
Parsing TAP test result [/net/www/ci/fedbiomed/jobs/fedbiomed-integration-linux/builds/112/tap-master-files/report.filtered.patched.tap].
TAP Reports Processing: FINISH
Finished: SUCCESS
Dear user, please keep the lign below when submitting an issue: