Enable skipping frozen weights when using `Model.get_weights` and `Model.set_weights`.
Currently, the specifications of Model
gradients and weights getter (respectively setter) are the following:
-
get_weights
: return all model weights, wrapped as aVector
-
compute_batch_gradients
: return gradients wrt trainable model weights, wrapped as aVector
This is generally not an issue, and feels natural to both ignore frozen weights when computing gradients (otherwise they would not be frozen) and have a way to access (and thus, share and/or save) all model weights, including frozen ones.
However, this will cause Optimizer
to fail to refine gradients into model updates when some weights are frozen when some Regularizer
plug-ins and/or weight decay are used. As a matter of fact, both previous features compute gradient-correction terms based on the full set of weights, that will fail to be combined with the gradients-based-updates Vector
that only matches trainable weights.
I therefore suggest to implement one of the following minor API changes:
- A. Add a
trainable_only: bool = False
argument toModel.get_weights
andModel.set_weights
and use it appropriately. - B. Add a
Model.get_trainable_weights
method and use it as part ofOptimizer
.