training arguments set using `experiment.set_training_args` are ignored during training
Steps to reproduce
- in the 101 notebook, after the experiment has been created but before it is run, change anything in the training arguments using the
exp.set_training_args()
method. - run the experiment
- you will see that the changes you made to the training arguments were not communicated to the nodes
More details
This is due to the fact that:
- the
experiment.job
keeps a duplicate of the training arguments as its own member - the training arguments that are actually sent to the nodes are those inside the job
- the
experiment.set_training_args
function does not propagate the changes to the job
A quick fix would be to propagate the changes to the job. A much better fix would be to avoid duplicating the information about training arguments (I believe that only the Experiment should have them, but this can be discussed).