evaluation_interval is confusing
The parameters evaluation_interval
is quite confusing, or at least requires some effort to compute, which is time-consuming, and error-prone.
It is also a bit annoying because the number of samples in the training set is not directly accessible (one needs to grep -c train preprocessed_param.csv
to find it). Still bad UX.
This issue can be easily solved by improving the doc (saying its in number of batch, and not samples, it's currently unclear), and by providing the number of samples in each split after the splitting into train/validation/test. More importantly, adding a mutually exclusive parameter like evaluation_steps
would improve greatly the UX IMHO.
Currently, when we want to change the batch size, and keep the same number of evaluations, we need to recompute this value to keep the number of steps constant. Obviously this is not a complex calculation, but still quite annoying to do.
I think having both parameters being mutually exclusive seems the best solution. For instance, there are 640k samples in the training set, and I want 10 evaluation steps, or an evaluation every 1000 batches, and I have a batch size of 64 :
evaluation_interval: 1000 # 640000 / 10 / 64
batch_size: 64
and
evaluation_steps: 10 # 640000 / 1000 / 64
batch_size: 64
The second one feels more natural for my use case (I want to change the batch size but keep the number of evaluations), but probably others prefer to think in terms of batches (they want to change the batch size, but keep evaluation after a given number of batches).