Error during segmentation training
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 474, in run
return task._orig_run(*args, **kwargs)
File "/usr/src/app/apps/core/tasks.py", line 221, in segtrain
raise e
File "/usr/src/app/apps/core/tasks.py", line 211, in segtrain
shutil.copy(best_version, modelpath)
File "/usr/local/lib/python3.7/shutil.py", line 248, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/local/lib/python3.7/shutil.py", line 120, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/app/media/models/17/version_1.mlmodel'
I think what is happening is there is a limit to the number of versions we keep, and in this case the best detected version is inferior to the one we keep.
We should disable the maximum number of versions for models but also devise a strategy to clean them up.