Initial support for dnadna train --resume
Now, if you run dnadna train --resume <path/to/training/config.yml
it
will resume from the last "last_checkpoint" model file. The capability
exists also to resume from the last best net, but it's not integrated
into the CLI yet.
No tests for this yet (it is difficult to test) but I've done several tests now of stopping and resuming training, and it appears to work correctly.
I don't know if we'll have enough time to test this throughly enough to make it a 1.0 feature. Or we could go ahead and add it and warn that it is "experimental".
This is also enhanced by !127 (merged) which makes interrupting a training run a bit nicer.
Part of the discussion in #43.