Validate `model_arguments` with the training plan created on the researcher side
Regarding the longer term issue, it still exists though it is probably out of this bug's scope:
for longer term, we should have conventions to robustify training plans (for example: always assign default values to model_args fields or always check if the field exists before accessing it). Same issue of robustness exists with
training_args
and may happen when instantiating/training on the node.
Example of 2. : if we change to model_args = {}
in monai-2d-image-classification.ipynb
the notebook fails while instantiating Experiment()
with an ugly backtrace of:
--------------------
Fed-BioMed researcher stopped due to unknown error:
'num_class'
More details in the backtrace extract below
--------------------
Traceback (most recent call last):
File "/home/mvesin/GIT/fedbiomed/fedbiomed/fedbiomed/researcher/experiment.py", line 63, in payload
ret = function(*args, **kwargs)
File "/home/mvesin/GIT/fedbiomed/fedbiomed/fedbiomed/researcher/experiment.py", line 1401, in set_job
self._job = Job(reqs=self._reqs,
File "/home/mvesin/GIT/fedbiomed/fedbiomed/fedbiomed/researcher/job.py", line 144, in __init__
self._training_plan.post_init(model_args={} if self._model_args is None else self._model_args,
File "/home/mvesin/GIT/fedbiomed/fedbiomed/fedbiomed/common/training_plans/_torchnn.py", line 157, in post_init
self._configure_model_and_optimizer()
File "/home/mvesin/GIT/fedbiomed/fedbiomed/fedbiomed/common/training_plans/_torchnn.py", line 282, in _configure_model_and_optimizer
self._model = self.init_model()
KeyError: 'num_class'
--------------------
We could try to robustify model_args access by patching training plans, eg for pytorch:
def model_args(self, key: str = None) -> Dict:
"""Retrieves model args
Returns:
Model arguments arguments
"""
if key is None:
return self._model_args
else:
if key in self._model_args:
return self._model_args[key]
else:
raise FedbiomedTrainingPlanError(f"{ErrorNumbers.FB605.value}: bad model_arg {key}")
... and then replace accesses to model_args eg in monai-2d-image-classification.ipynb
:
def init_model(self):
model = DenseNet121(spatial_dims=2, in_channels=1,
out_channels = self.model_args("num_class"))
return model
... so that with model_args = {}
instantiate Experiment()
fails more gently with a:
--------------------
Fed-BioMed researcher stopped due to exception:
FB605: training plan error: bad model_arg num_class
--------------------