Validation step is too slow
During the training procedure, the number of worker for validation is reduced proportionally to the size ratio of the training and validation set.
The initial intention was to increase the speed of the validation pass, but in practice, it slows the validation a lot.
A few solutions :
- Remove this part of the code, and just reuse the same number of worker as for the training.
- Add an option for a different number of workers for the validation step.
- There is an option starting from pytorch 1.7 called "persistent_worker" that might help here